[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:51:52 dist-test-slave-4mtw krb5kdc[2417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:51:52 dist-test-slave-4mtw krb5kdc[2417](info): set up 2 sockets
Jun 20 22:51:52 dist-test-slave-4mtw krb5kdc[2417](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 22:51:53.788975 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:51:53 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:51:53.897287 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37605
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:37605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:53.974213 2430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:53.974431 2430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:53.974460 2430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:53.975592 2430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:51:53.975637 2430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:53.975656 2430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:53.975672 2430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:51:53.975687 2430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:51:53.977160 2430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:37605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:53.977401 2430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:53.977640 2430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:53.982630 2438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:53.982612 2436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:53.982646 2430 server_base.cc:1034] running on GCE node
W20260620 22:51:53.982606 2435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:53.983011 2430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:53.983312 2430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:53.984472 2430 hybrid_clock.cc:648] HybridClock initialized: now 1781995913984447 us; error 39 us; skew 500 ppm
Jun 20 22:51:53 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995913, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:53.986799 2430 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:51:53.987210 2430 webserver.cc:460] Webserver started at http://127.2.90.190:34943/ using document root <none> and password file <none>
I20260620 22:51:53.987427 2430 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:53.987485 2430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:53.987610 2430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:53.988639 2430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84"
format_stamp: "Formatted at 2026-06-20 22:51:53 on dist-test-slave-4mtw"
server_key: "bd5e230f15a24ebde4a93f2b8604bce5"
server_key_iv: "8e0168c563e0517ff305f90623f148bc"
server_key_version: "encryptionkey@0"
I20260620 22:51:53.989008 2430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84"
format_stamp: "Formatted at 2026-06-20 22:51:53 on dist-test-slave-4mtw"
server_key: "bd5e230f15a24ebde4a93f2b8604bce5"
server_key_iv: "8e0168c563e0517ff305f90623f148bc"
server_key_version: "encryptionkey@0"
I20260620 22:51:53.990324 2430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:51:53.990998 2445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:53.991146 2430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:51:53.991227 2430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84"
format_stamp: "Formatted at 2026-06-20 22:51:53 on dist-test-slave-4mtw"
server_key: "bd5e230f15a24ebde4a93f2b8604bce5"
server_key_iv: "8e0168c563e0517ff305f90623f148bc"
server_key_version: "encryptionkey@0"
I20260620 22:51:53.991293 2430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:54.021188 2430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:54.023168 2430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:54.023317 2430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:54.027307 2430 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:37605
I20260620 22:51:54.027336 2497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:37605 every 8 connection(s)
I20260620 22:51:54.027652 2430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:51:54.028455 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:54.030628 2498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84: Bootstrap starting.
I20260620 22:51:54.031127 2498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:54.031358 2498 log.cc:824] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:54.031499 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 2430
I20260620 22:51:54.031586 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:51:54.032033 2498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84: No bootstrap required, opened a new log
I20260620 22:51:54.032182 2410 external_mini_cluster.cc:1428] Setting key 977409253f886497ce831501ac2e96cf
I20260620 22:51:54.033972 2498 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37605 } }
I20260620 22:51:54.034148 2498 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:54.034183 2498 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b29a3d9c3b4f0fbd934573c9c61c84, State: Initialized, Role: FOLLOWER
I20260620 22:51:54.034312 2498 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37605 } }
I20260620 22:51:54.034387 2498 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:51:54.034435 2498 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:51:54.034480 2498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:54.035516 2498 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37605 } }
I20260620 22:51:54.035627 2498 leader_election.cc:302] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b29a3d9c3b4f0fbd934573c9c61c84; no voters:
I20260620 22:51:54.035776 2498 leader_election.cc:288] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:51:54.035809 2503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:54.035934 2503 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [term 1 LEADER]: Becoming Leader. State: Replica: e8b29a3d9c3b4f0fbd934573c9c61c84, State: Running, Role: LEADER
I20260620 22:51:54.036015 2498 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:51:54.036019 2503 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37605 } }
I20260620 22:51:54.036435 2504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37605 } } }
I20260620 22:51:54.036506 2504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:54.036651 2505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b29a3d9c3b4f0fbd934573c9c61c84. Latest consensus state: current_term: 1 leader_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b29a3d9c3b4f0fbd934573c9c61c84" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37605 } } }
I20260620 22:51:54.036727 2505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84 [sys.catalog]: This master's current role is: LEADER
W20260620 22:51:54.037618 2519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:51:54.037662 2519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:51:54.037722 2511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:51:54.038177 2511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:51:54.039629 2511 catalog_manager.cc:1347] Generated new cluster ID: ed067b7d00354b298a39d11b868d1f7e
I20260620 22:51:54.039680 2511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:54.050063 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.033030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.2.90.190:37605)
0620 22:51:54.034282 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:54.034291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:54.034330 (+ 39us) server_negotiation.cc:407] Connection header received
0620 22:51:54.035701 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:54.035709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:54.035762 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:54.035919 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:54.038542 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.039733 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.041316 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.041893 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.042040 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.046243 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:54.046248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:54.046248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:54.046259 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:54.048128 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.048617 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.048620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.048620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.048744 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.049008 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.049008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.049009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.049291 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:54.049431 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:54.049697 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 22:51:54.049826 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":81,"threads_started":1}
I20260620 22:51:54.054292 2511 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:51:54.054993 2511 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:51:54.060582 2511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8b29a3d9c3b4f0fbd934573c9c61c84: Generated new TSK 0
I20260620 22:51:54.060778 2511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:51:54.087383 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37605
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:54.185540 2524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:54.185729 2524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:54.185750 2524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:54.186852 2524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:54.186893 2524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:54.186932 2524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:51:54.188367 2524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:54.188593 2524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:54.188822 2524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:54.193289 2530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:54.193305 2532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:54.193482 2524 server_base.cc:1034] running on GCE node
W20260620 22:51:54.193568 2529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:54.193734 2524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:54.193985 2524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:54.195129 2524 hybrid_clock.cc:648] HybridClock initialized: now 1781995914195116 us; error 28 us; skew 500 ppm
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:54.197409 2524 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:51:54.197767 2524 webserver.cc:460] Webserver started at http://127.2.90.129:35063/ using document root <none> and password file <none>
I20260620 22:51:54.197929 2524 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:54.197974 2524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:54.198061 2524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:54.198987 2524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "463e9fc757184bb6b72f38c77456e94b"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "b80481f0fb9daac192ee2da00c567f92"
server_key_iv: "217797192104eb43fc5718f1ad7c4bc9"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.199267 2524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "463e9fc757184bb6b72f38c77456e94b"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "b80481f0fb9daac192ee2da00c567f92"
server_key_iv: "217797192104eb43fc5718f1ad7c4bc9"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.200328 2524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:51:54.200930 2539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.201076 2524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:51:54.201151 2524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "463e9fc757184bb6b72f38c77456e94b"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "b80481f0fb9daac192ee2da00c567f92"
server_key_iv: "217797192104eb43fc5718f1ad7c4bc9"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.201203 2524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:54.219072 2524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:54.220778 2524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:54.220887 2524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:54.221074 2524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:51:54.221320 2524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:54.221349 2524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.221374 2524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:54.221387 2524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.227581 2524 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:41791
I20260620 22:51:54.227648 2652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:41791 every 8 connection(s)
I20260620 22:51:54.227864 2524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:51:54.232173 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 2524
I20260620 22:51:54.232275 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:51:54.232429 2410 external_mini_cluster.cc:1428] Setting key 922eabdad1b780ebb8c4078a267c55b8
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:54.236773 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.228446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:45653 (local address 127.2.90.190:37605)
0620 22:51:54.228486 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:54.228488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:54.228706 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:51:54.229197 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:54.229198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:54.229214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:54.229265 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:54.230170 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.230659 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.231610 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.231801 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.231854 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.234019 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:54.234021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:54.234021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:54.234027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:54.235428 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.235795 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.235796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.235797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.235855 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.236072 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.236073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.236074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.236196 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:54.236264 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:54.236578 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 22:51:54.236655 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:51:54.236876 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.228416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37605 (local address 127.2.90.129:45653)
0620 22:51:54.228579 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:54.228612 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:54.229103 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:54.229333 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:54.229337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:54.229887 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:54.230078 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:54.230083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.230736 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.230737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:54.231531 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:54.231533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.231865 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.231866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:54.231983 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.232461 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:54.232469 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:54.233933 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:54.235510 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:54.235515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:54.235517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:54.235709 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:54.235918 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:54.235919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:54.235919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:54.236018 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:54.236252 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:54.236256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:54.236415 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 22:51:54.236540 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 22:51:54.236658 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260620 22:51:54.237323 2653 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37605
I20260620 22:51:54.237424 2653 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:54.237629 2653 heartbeater.cc:511] Master 127.2.90.190:37605 requested a full tablet report, sending...
I20260620 22:51:54.238154 2462 ts_manager.cc:194] Registered new tserver with Master: 463e9fc757184bb6b72f38c77456e94b (127.2.90.129:41791)
I20260620 22:51:54.239001 2462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:45653
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:51:54.255813 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37605
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:54.332494 2658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:54.332696 2658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:54.332726 2658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:54.333793 2658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:54.333827 2658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:54.333850 2658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:51:54.335203 2658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:54.335372 2658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:54.335546 2658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:54.339941 2666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:54.339946 2664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:54.339951 2663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:54.340270 2658 server_base.cc:1034] running on GCE node
I20260620 22:51:54.340441 2658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:54.340658 2658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:54.341825 2658 hybrid_clock.cc:648] HybridClock initialized: now 1781995914341813 us; error 28 us; skew 500 ppm
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:54.343983 2658 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:51:54.344376 2658 webserver.cc:460] Webserver started at http://127.2.90.130:42907/ using document root <none> and password file <none>
I20260620 22:51:54.344547 2658 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:54.344583 2658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:54.344671 2658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:54.345624 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "ee12399e3d494e4e822b52a3cbeb731b"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "52989def64eae47dfb055ebf91d900d3"
server_key_iv: "5ab60d18cf2d9c7eded0f18feee517fd"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.345891 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "ee12399e3d494e4e822b52a3cbeb731b"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "52989def64eae47dfb055ebf91d900d3"
server_key_iv: "5ab60d18cf2d9c7eded0f18feee517fd"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.347015 2658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:51:54.347608 2673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.347790 2658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:51:54.347868 2658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "ee12399e3d494e4e822b52a3cbeb731b"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "52989def64eae47dfb055ebf91d900d3"
server_key_iv: "5ab60d18cf2d9c7eded0f18feee517fd"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.347920 2658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:54.373451 2658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:54.375229 2658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:54.375342 2658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:54.375535 2658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:51:54.375788 2658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:54.375818 2658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.375838 2658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:54.375852 2658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.381597 2658 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:40381
I20260620 22:51:54.381637 2786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:40381 every 8 connection(s)
I20260620 22:51:54.381920 2658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:54.389801 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.382566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:46595 (local address 127.2.90.190:37605)
0620 22:51:54.382619 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:54.382621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:54.382810 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:51:54.383259 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:54.383260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:54.383269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:54.383319 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:54.384176 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.384702 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.385612 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.385788 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.385830 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.387689 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:54.387691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:54.387691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:54.387699 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:54.388790 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.389118 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.389119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.389119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.389163 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.389327 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.389327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.389328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.389439 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:54.389484 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:54.389678 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 22:51:54.389735 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:51:54.390017 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 2658
I20260620 22:51:54.390100 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:51:54.390174 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.382551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37605 (local address 127.2.90.130:46595)
0620 22:51:54.382698 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:54.382733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:54.383187 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:54.383373 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:54.383377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:54.383902 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:54.384095 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:54.384099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.384759 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.384759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:54.385555 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:54.385556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.385838 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.385839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:54.385929 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.386331 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:54.386338 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:54.387607 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:54.388853 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:54.388858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:54.388860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:54.389064 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:54.389204 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:54.389204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:54.389204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:54.389288 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:54.389481 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:54.389483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:54.389588 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 22:51:54.389662 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:51:54.389772 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260620 22:51:54.390257 2410 external_mini_cluster.cc:1428] Setting key 78b2b7c54ec0ce57d12f7495bbf32af9
I20260620 22:51:54.390717 2787 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37605
I20260620 22:51:54.390810 2787 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:54.390995 2787 heartbeater.cc:511] Master 127.2.90.190:37605 requested a full tablet report, sending...
I20260620 22:51:54.391340 2462 ts_manager.cc:194] Registered new tserver with Master: ee12399e3d494e4e822b52a3cbeb731b (127.2.90.130:40381)
I20260620 22:51:54.391675 2462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:46595
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:51:54.411070 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37605
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:54.490475 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:54.490666 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:54.490687 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:54.491788 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:54.491829 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:54.491856 2792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:51:54.493314 2792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37605
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:54.493515 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:54.493712 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:54.498247 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:54.498320 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:54.498266 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:54.498391 2792 server_base.cc:1034] running on GCE node
I20260620 22:51:54.498553 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:54.498757 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:54.499897 2792 hybrid_clock.cc:648] HybridClock initialized: now 1781995914499877 us; error 29 us; skew 500 ppm
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:54.502094 2792 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:51:54.502463 2792 webserver.cc:460] Webserver started at http://127.2.90.131:43383/ using document root <none> and password file <none>
I20260620 22:51:54.502643 2792 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:54.502688 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:54.502815 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:54.503793 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "cf943f6f65b24cbe81ed7a5607718b4e"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "93b17f4df2c71a931e43d3cfd376e431"
server_key_iv: "dd94c626c8c53d7d87346db0b9d7350c"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.504122 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "cf943f6f65b24cbe81ed7a5607718b4e"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "93b17f4df2c71a931e43d3cfd376e431"
server_key_iv: "dd94c626c8c53d7d87346db0b9d7350c"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.505362 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:51:54.505991 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.506126 2792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:51:54.506202 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "cf943f6f65b24cbe81ed7a5607718b4e"
format_stamp: "Formatted at 2026-06-20 22:51:54 on dist-test-slave-4mtw"
server_key: "93b17f4df2c71a931e43d3cfd376e431"
server_key_iv: "dd94c626c8c53d7d87346db0b9d7350c"
server_key_version: "encryptionkey@0"
I20260620 22:51:54.506259 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:54.515775 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:54.517433 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:54.517560 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:54.517751 2792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:51:54.518014 2792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:54.518045 2792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.518072 2792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:54.518090 2792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:54.524011 2792 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:38675
I20260620 22:51:54.524102 2920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:38675 every 8 connection(s)
I20260620 22:51:54.524358 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:51:54.524828 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 2792
I20260620 22:51:54.524907 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:51:54.525080 2410 external_mini_cluster.cc:1428] Setting key b99b5567d8ed30b93469f9e5f95cce1b
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:54.532843 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.525004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:52577 (local address 127.2.90.190:37605)
0620 22:51:54.525054 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:54.525056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:54.525286 (+ 230us) server_negotiation.cc:407] Connection header received
0620 22:51:54.525838 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:54.525839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:54.525850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:54.525908 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:54.526795 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.527263 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.528189 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.528377 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.528419 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.530379 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:54.530380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:54.530381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:54.530387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:54.531521 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.532004 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.532005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.532006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.532069 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.532338 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.532339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.532339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.532438 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:54.532487 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:54.532705 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 22:51:54.532775 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:51:54.533023 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.524992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37605 (local address 127.2.90.131:52577)
0620 22:51:54.525165 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:54.525206 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:54.525742 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:54.525980 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:54.525984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:54.526545 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:54.526726 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:54.526730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.527357 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.527357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:54.528132 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:54.528134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.528434 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.528435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:54.528526 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.529009 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:54.529019 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:54.530298 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:54.531615 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:54.531621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:54.531623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:54.531931 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:54.532127 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:54.532128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:54.532128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:54.532267 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:54.532476 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:54.532479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:54.532587 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 22:51:54.532662 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:51:54.532814 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260620 22:51:54.533402 2921 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37605
I20260620 22:51:54.533488 2921 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:54.533653 2921 heartbeater.cc:511] Master 127.2.90.190:37605 requested a full tablet report, sending...
I20260620 22:51:54.533969 2462 ts_manager.cc:194] Registered new tserver with Master: cf943f6f65b24cbe81ed7a5607718b4e (127.2.90.131:38675)
I20260620 22:51:54.534314 2462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:52577
I20260620 22:51:54.536934 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:51:54.557828 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.550999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.2.90.129:41791)
0620 22:51:54.551199 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:54.551203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:54.551215 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:51:54.551240 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:54.551242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:54.551255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:54.551325 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:54.551826 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.552560 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.553235 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.553562 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.553612 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.555577 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:54.555585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:54.555585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:54.555591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:54.556977 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.557258 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.557260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.557260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.557317 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.557445 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.557446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.557446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.557557 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:54.557663 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:54.557697 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:51:54.557750 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":116,"threads_started":1}
W20260620 22:51:54.558223 2567 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38794
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:51:54.578617 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:54.572533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.2.90.129:41791)
0620 22:51:54.572597 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:54.572599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:54.572655 (+ 56us) server_negotiation.cc:407] Connection header received
0620 22:51:54.572753 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:54.572754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:54.572769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:54.572816 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:54.573309 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.573789 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.574469 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:54.574656 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:54.574702 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:54.576633 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:54.576634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:54.576634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:54.576640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:54.577867 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.578123 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.578123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.578124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.578167 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:54.578286 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:54.578287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:54.578287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:54.578363 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:54.578441 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:54.578495 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:51:54.578551 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:51:54.579087 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 2524
I20260620 22:51:54.583736 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 2658
I20260620 22:51:54.588039 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 2792
I20260620 22:51:54.591722 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 2430
2026-06-20T22:51:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1757 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2935](info): set up 2 sockets
Jun 20 22:51:54 dist-test-slave-4mtw krb5kdc[2935](info): commencing operation
krb5kdc: starting...
W20260620 22:51:55.539937 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:51:55 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:51:55.644371 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:40787
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:55.720822 2948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:55.720997 2948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:55.721017 2948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:55.722072 2948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:51:55.722105 2948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:55.722117 2948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:55.722128 2948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:51:55.722144 2948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:51:55.723512 2948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:40787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:55.723680 2948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:55.723840 2948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:55.728226 2953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:55.728291 2956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:55.728240 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:55.728376 2948 server_base.cc:1034] running on GCE node
I20260620 22:51:55.728598 2948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:55.728880 2948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:55.730023 2948 hybrid_clock.cc:648] HybridClock initialized: now 1781995915730010 us; error 28 us; skew 500 ppm
Jun 20 22:51:55 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:55.732232 2948 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:51:55.732654 2948 webserver.cc:460] Webserver started at http://127.2.90.190:44035/ using document root <none> and password file <none>
I20260620 22:51:55.732887 2948 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:55.732933 2948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:55.733052 2948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:55.734150 2948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "508a62014d2e4450aa1524758162fc2a"
format_stamp: "Formatted at 2026-06-20 22:51:55 on dist-test-slave-4mtw"
server_key: "567f7c20a44620e0283f6a7810254219"
server_key_iv: "5f5e13ae394fbebb6e543064a5d6062d"
server_key_version: "encryptionkey@0"
I20260620 22:51:55.734484 2948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "508a62014d2e4450aa1524758162fc2a"
format_stamp: "Formatted at 2026-06-20 22:51:55 on dist-test-slave-4mtw"
server_key: "567f7c20a44620e0283f6a7810254219"
server_key_iv: "5f5e13ae394fbebb6e543064a5d6062d"
server_key_version: "encryptionkey@0"
I20260620 22:51:55.735706 2948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:51:55.736373 2963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:55.736559 2948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:51:55.736639 2948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "508a62014d2e4450aa1524758162fc2a"
format_stamp: "Formatted at 2026-06-20 22:51:55 on dist-test-slave-4mtw"
server_key: "567f7c20a44620e0283f6a7810254219"
server_key_iv: "5f5e13ae394fbebb6e543064a5d6062d"
server_key_version: "encryptionkey@0"
I20260620 22:51:55.736732 2948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:55.750042 2948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:55.751878 2948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:55.752013 2948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:55.755841 2948 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:40787
I20260620 22:51:55.755884 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:40787 every 8 connection(s)
I20260620 22:51:55.756209 2948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:51:55.757050 3016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:55.758123 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 2948
I20260620 22:51:55.758211 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:51:55.758368 2410 external_mini_cluster.cc:1428] Setting key 7c55560a8e6c0aca021540523a0f6833
I20260620 22:51:55.759936 3016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a: Bootstrap starting.
I20260620 22:51:55.760565 3016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:55.760893 3016 log.cc:824] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:55.761582 3016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a: No bootstrap required, opened a new log
I20260620 22:51:55.763711 3016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508a62014d2e4450aa1524758162fc2a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40787 } }
I20260620 22:51:55.763845 3016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:55.763866 3016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508a62014d2e4450aa1524758162fc2a, State: Initialized, Role: FOLLOWER
I20260620 22:51:55.763979 3016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508a62014d2e4450aa1524758162fc2a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40787 } }
I20260620 22:51:55.764050 3016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:51:55.764074 3016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:51:55.764119 3016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:55.764729 3016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508a62014d2e4450aa1524758162fc2a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40787 } }
I20260620 22:51:55.764835 3016 leader_election.cc:302] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 508a62014d2e4450aa1524758162fc2a; no voters:
I20260620 22:51:55.764999 3016 leader_election.cc:288] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 20 22:51:55 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:55.765035 3021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:55.765169 3021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [term 1 LEADER]: Becoming Leader. State: Replica: 508a62014d2e4450aa1524758162fc2a, State: Running, Role: LEADER
I20260620 22:51:55.765254 3016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:51:55.765259 3021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508a62014d2e4450aa1524758162fc2a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40787 } }
I20260620 22:51:55.765585 3021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 508a62014d2e4450aa1524758162fc2a. Latest consensus state: current_term: 1 leader_uuid: "508a62014d2e4450aa1524758162fc2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508a62014d2e4450aa1524758162fc2a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40787 } } }
I20260620 22:51:55.765666 3021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:55.765858 3026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:51:55.765897 3023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "508a62014d2e4450aa1524758162fc2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "508a62014d2e4450aa1524758162fc2a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40787 } } }
I20260620 22:51:55.765954 3023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:55.766328 3026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:51:55.767880 3026 catalog_manager.cc:1347] Generated new cluster ID: 8ec46c699b294463ad7b3aaa4feb7546
I20260620 22:51:55.767926 3026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:51:55.769030 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:55.758896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.2.90.190:40787)
0620 22:51:55.759073 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:55.759083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:55.759117 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:51:55.759615 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:55.759624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:55.759687 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:55.759864 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:55.760404 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:55.761706 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:55.762401 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:55.763020 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:55.763137 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:55.765595 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:55.765600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:55.765600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:55.765613 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:55.767560 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:55.767944 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:55.767947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:55.767947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:55.768051 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:55.768226 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:55.768227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:55.768227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:55.768457 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:55.768576 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:55.768706 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 22:51:55.768832 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260620 22:51:55.782294 3026 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:51:55.783013 3026 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:51:55.793915 3026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 508a62014d2e4450aa1524758162fc2a: Generated new TSK 0
I20260620 22:51:55.794126 3026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:51:55.827152 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:40787
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:55.906277 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:55.906466 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:55.906497 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:55.907594 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:55.907631 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:55.907666 3042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:51:55.909130 3042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:40787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:55.909361 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:55.909552 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:55.914031 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:55.914080 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:55.914177 3042 server_base.cc:1034] running on GCE node
W20260620 22:51:55.914080 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:55.914458 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:55.914692 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:55.915843 3042 hybrid_clock.cc:648] HybridClock initialized: now 1781995915915828 us; error 28 us; skew 500 ppm
Jun 20 22:51:55 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:55.918145 3042 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:51:55.918561 3042 webserver.cc:460] Webserver started at http://127.2.90.129:42601/ using document root <none> and password file <none>
I20260620 22:51:55.918751 3042 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:55.918795 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:55.918911 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:55.919893 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "1798ab5d07b44d65970968f4a431e3c9"
format_stamp: "Formatted at 2026-06-20 22:51:55 on dist-test-slave-4mtw"
server_key: "31c2fa9b7ca608bcfd86ba0cdb56abb4"
server_key_iv: "013b3fa01f67fc43a12d6181d1eba5ca"
server_key_version: "encryptionkey@0"
I20260620 22:51:55.920200 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "1798ab5d07b44d65970968f4a431e3c9"
format_stamp: "Formatted at 2026-06-20 22:51:55 on dist-test-slave-4mtw"
server_key: "31c2fa9b7ca608bcfd86ba0cdb56abb4"
server_key_iv: "013b3fa01f67fc43a12d6181d1eba5ca"
server_key_version: "encryptionkey@0"
I20260620 22:51:55.921443 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:51:55.922043 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:55.922178 3042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:51:55.922278 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "1798ab5d07b44d65970968f4a431e3c9"
format_stamp: "Formatted at 2026-06-20 22:51:55 on dist-test-slave-4mtw"
server_key: "31c2fa9b7ca608bcfd86ba0cdb56abb4"
server_key_iv: "013b3fa01f67fc43a12d6181d1eba5ca"
server_key_version: "encryptionkey@0"
I20260620 22:51:55.922333 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:55.930151 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:55.931699 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:55.931818 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:55.932001 3042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:51:55.932269 3042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:55.932301 3042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:55.932328 3042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:55.932353 3042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:55.937770 3042 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:42439
I20260620 22:51:55.937819 3170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:42439 every 8 connection(s)
I20260620 22:51:55.938091 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:51:55.941680 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3042
I20260620 22:51:55.941751 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:51:55.941876 2410 external_mini_cluster.cc:1428] Setting key 1be8d0b1568c2296d7ac9026f17c819e
Jun 20 22:51:55 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:55.946549 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:55.938699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:47923 (local address 127.2.90.190:40787)
0620 22:51:55.938750 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:55.938752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:55.938936 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:51:55.939837 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:55.939839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:55.939848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:55.939904 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:55.940453 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:55.940964 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:55.941912 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:55.942176 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:55.942232 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:55.944169 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:55.944171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:55.944171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:55.944178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:55.945402 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:55.945726 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:55.945727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:55.945727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:55.945792 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:55.945988 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:55.945989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:55.945989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:55.946093 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:55.946146 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:55.946387 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 22:51:55.946464 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:51:55.946693 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:55.938680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:40787 (local address 127.2.90.129:47923)
0620 22:51:55.938827 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:55.938864 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:55.939750 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:55.939953 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:55.939956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:55.940180 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:55.940389 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:55.940393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:55.941018 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:55.941019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:55.941842 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:55.941845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:55.942295 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:55.942296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:55.942401 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:55.942829 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:55.942837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:55.944088 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:55.945463 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:55.945469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:55.945471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:55.945664 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:55.945845 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:55.945845 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:55.945846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:55.945936 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:55.946143 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:55.946149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:55.946276 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:51:55.946358 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:51:55.946494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260620 22:51:55.947077 3171 heartbeater.cc:347] Connected to a master server at 127.2.90.190:40787
I20260620 22:51:55.947149 3171 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:55.947301 3171 heartbeater.cc:511] Master 127.2.90.190:40787 requested a full tablet report, sending...
I20260620 22:51:55.947785 2980 ts_manager.cc:194] Registered new tserver with Master: 1798ab5d07b44d65970968f4a431e3c9 (127.2.90.129:42439)
I20260620 22:51:55.948664 2980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:47923
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:51:55.963451 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:40787
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:56.040776 3176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:56.040959 3176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:56.040982 3176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:56.042098 3176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:56.042137 3176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:56.042162 3176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:51:56.043613 3176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:40787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:56.043823 3176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:56.044024 3176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:56.048493 3184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:56.048594 3176 server_base.cc:1034] running on GCE node
W20260620 22:51:56.048513 3182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:56.048514 3181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:56.048943 3176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:56.049158 3176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:56.050292 3176 hybrid_clock.cc:648] HybridClock initialized: now 1781995916050282 us; error 26 us; skew 500 ppm
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:56.052551 3176 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:51:56.052951 3176 webserver.cc:460] Webserver started at http://127.2.90.130:46089/ using document root <none> and password file <none>
I20260620 22:51:56.053133 3176 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:56.053169 3176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:56.053253 3176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:56.054169 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "095090a6b8f345a7b66972e4c7ab9a12"
format_stamp: "Formatted at 2026-06-20 22:51:56 on dist-test-slave-4mtw"
server_key: "f00868e3cfc225312e5f96b2308106ff"
server_key_iv: "f47debc6cfa8b7ea514bb9570f5847bc"
server_key_version: "encryptionkey@0"
I20260620 22:51:56.054440 3176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "095090a6b8f345a7b66972e4c7ab9a12"
format_stamp: "Formatted at 2026-06-20 22:51:56 on dist-test-slave-4mtw"
server_key: "f00868e3cfc225312e5f96b2308106ff"
server_key_iv: "f47debc6cfa8b7ea514bb9570f5847bc"
server_key_version: "encryptionkey@0"
I20260620 22:51:56.055526 3176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:51:56.056126 3191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:56.056267 3176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:51:56.056363 3176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "095090a6b8f345a7b66972e4c7ab9a12"
format_stamp: "Formatted at 2026-06-20 22:51:56 on dist-test-slave-4mtw"
server_key: "f00868e3cfc225312e5f96b2308106ff"
server_key_iv: "f47debc6cfa8b7ea514bb9570f5847bc"
server_key_version: "encryptionkey@0"
I20260620 22:51:56.056421 3176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:56.066896 3176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:56.068480 3176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:56.068596 3176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:56.068822 3176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:51:56.069084 3176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:56.069115 3176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:56.069141 3176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:56.069164 3176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:56.074209 3176 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:35607
I20260620 22:51:56.074265 3304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:35607 every 8 connection(s)
I20260620 22:51:56.074518 3176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:51:56.077348 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3176
I20260620 22:51:56.077451 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:51:56.077596 2410 external_mini_cluster.cc:1428] Setting key da2242c9e5e80f1b0475bc981aab2cd5
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:56.083602 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.075105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:54853 (local address 127.2.90.190:40787)
0620 22:51:56.075158 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.075160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.075339 (+ 179us) server_negotiation.cc:407] Connection header received
0620 22:51:56.076162 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.076163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.076172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.076219 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:56.076769 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.077300 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.078429 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.078692 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.078758 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.080970 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:56.080972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:56.080972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:56.080980 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:56.082355 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.082784 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.082784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.082785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.082828 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.083059 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.083059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.083060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.083150 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:56.083194 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.083478 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.083541 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:51:56.083832 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.075102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:40787 (local address 127.2.90.130:54853)
0620 22:51:56.075235 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.075265 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.076086 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.076271 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.076274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.076506 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:56.076699 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.076704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.077371 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.077372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.078358 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.078361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.078769 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.078770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.078891 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.079368 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:56.079379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:56.080852 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:56.082428 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:56.082434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:56.082436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:56.082712 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:56.082880 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:56.082881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:56.082881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:56.083004 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:56.083201 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:56.083206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:56.083362 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.083470 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.083618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260620 22:51:56.084328 3305 heartbeater.cc:347] Connected to a master server at 127.2.90.190:40787
I20260620 22:51:56.084420 3305 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:56.084575 3305 heartbeater.cc:511] Master 127.2.90.190:40787 requested a full tablet report, sending...
I20260620 22:51:56.084957 2980 ts_manager.cc:194] Registered new tserver with Master: 095090a6b8f345a7b66972e4c7ab9a12 (127.2.90.130:35607)
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
I20260620 22:51:56.085458 2980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:54853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:51:56.098742 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:40787
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:56.174973 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:56.175160 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:56.175184 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:56.176293 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:56.176331 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:56.176359 3310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:51:56.177812 3310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:40787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:56.178030 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:56.178234 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:56.182688 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:56.182713 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:56.182763 3310 server_base.cc:1034] running on GCE node
W20260620 22:51:56.182713 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:56.183033 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:56.183243 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:56.184381 3310 hybrid_clock.cc:648] HybridClock initialized: now 1781995916184365 us; error 26 us; skew 500 ppm
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:56.186645 3310 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:51:56.187026 3310 webserver.cc:460] Webserver started at http://127.2.90.131:46605/ using document root <none> and password file <none>
I20260620 22:51:56.187222 3310 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:56.187269 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:56.187390 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:56.188402 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "9ec7c47a190d44ffab6fee9c902d9c5b"
format_stamp: "Formatted at 2026-06-20 22:51:56 on dist-test-slave-4mtw"
server_key: "c9365fb2be6c2e08e3f0cb7b9a2dec55"
server_key_iv: "4a05bfecd95fb36ce919e41c1a22c019"
server_key_version: "encryptionkey@0"
I20260620 22:51:56.188788 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "9ec7c47a190d44ffab6fee9c902d9c5b"
format_stamp: "Formatted at 2026-06-20 22:51:56 on dist-test-slave-4mtw"
server_key: "c9365fb2be6c2e08e3f0cb7b9a2dec55"
server_key_iv: "4a05bfecd95fb36ce919e41c1a22c019"
server_key_version: "encryptionkey@0"
I20260620 22:51:56.189929 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:51:56.190532 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:56.190678 3310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:51:56.190769 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "9ec7c47a190d44ffab6fee9c902d9c5b"
format_stamp: "Formatted at 2026-06-20 22:51:56 on dist-test-slave-4mtw"
server_key: "c9365fb2be6c2e08e3f0cb7b9a2dec55"
server_key_iv: "4a05bfecd95fb36ce919e41c1a22c019"
server_key_version: "encryptionkey@0"
I20260620 22:51:56.190830 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:56.206931 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:56.208976 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:56.209103 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:56.209290 3310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:51:56.209575 3310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:56.209607 3310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:56.209635 3310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:56.209657 3310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:56.214694 3310 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:43597
I20260620 22:51:56.214752 3438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:43597 every 8 connection(s)
I20260620 22:51:56.215013 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:56.222468 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3310
I20260620 22:51:56.222571 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:51:56.222757 2410 external_mini_cluster.cc:1428] Setting key e31c759894460422c9dae151b007c67f
I20260620 22:51:56.223127 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.215625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:38813 (local address 127.2.90.190:40787)
0620 22:51:56.215670 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.215672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.215855 (+ 183us) server_negotiation.cc:407] Connection header received
0620 22:51:56.216741 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.216742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.216753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.216812 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:56.217335 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.217818 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.218727 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.218917 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.218978 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.220777 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:56.220778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:56.220778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:56.220785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:56.221862 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.222209 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.222210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.222211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.222276 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.222506 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.222506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.222507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.222649 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:56.222716 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.222967 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.223045 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:51:56.223317 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.215593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:40787 (local address 127.2.90.131:38813)
0620 22:51:56.215738 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.215777 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.216623 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.216867 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.216870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.217096 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:56.217271 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.217275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.217874 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.217875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.218642 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.218644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.218982 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.218982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.219070 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.219459 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:56.219466 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:56.220702 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:56.221925 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:56.221931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:56.221933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:56.222132 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:56.222355 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:56.222355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:56.222356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:56.222452 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:56.222705 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:56.222709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:56.222835 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.222936 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.223094 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 22:51:56.223716 3439 heartbeater.cc:347] Connected to a master server at 127.2.90.190:40787
I20260620 22:51:56.223794 3439 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:56.223956 3439 heartbeater.cc:511] Master 127.2.90.190:40787 requested a full tablet report, sending...
I20260620 22:51:56.224287 2979 ts_manager.cc:194] Registered new tserver with Master: 9ec7c47a190d44ffab6fee9c902d9c5b (127.2.90.131:43597)
I20260620 22:51:56.224646 2979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:38813
I20260620 22:51:56.234351 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:51:56.240955 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.235748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.2.90.190:40787)
0620 22:51:56.235795 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.235797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.235898 (+ 101us) server_negotiation.cc:407] Connection header received
0620 22:51:56.235959 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.235960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.235969 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.236015 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:56.236527 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.237051 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.237718 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.237916 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.237970 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.238947 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:56.238948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:56.238949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:56.238956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:56.239988 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.240259 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.240260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.240260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.240316 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.240467 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.240468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.240468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.240598 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:56.240704 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.240801 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.240871 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:51:56.242786 2979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:51:56.244174 2979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:51:56.249985 3448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.245984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35607 (local address 127.0.0.1:60104)
0620 22:51:56.246279 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.246314 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.246379 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.246585 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.246588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.246590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:56.246651 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.246652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.247905 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.247905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.248812 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.248814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.249693 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.249786 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.249802 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.249862 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.249913 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":58,"thread_start_us":51,"threads_started":1}
I20260620 22:51:56.250073 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.246070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.2.90.129:42439)
0620 22:51:56.246455 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.246459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.246468 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:51:56.246503 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.246505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.246517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.246583 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:56.246742 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.247816 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249239 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.249734 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249800 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.249854 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.249963 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.250004 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":87,"threads_started":1}
I20260620 22:51:56.250231 3451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.246167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:43597 (local address 127.0.0.1:37298)
0620 22:51:56.246690 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.246699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.246754 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.246893 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.246893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.246895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:56.246929 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.246930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.247820 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.247821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.249167 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.249169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.250032 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.250032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.250088 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.250091 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.250140 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.250173 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":43,"thread_start_us":64,"threads_started":1}
I20260620 22:51:56.250263 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.246087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.2.90.130:35607)
0620 22:51:56.246254 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.246259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.246452 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:51:56.246492 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.246494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.246508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.246569 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:56.246716 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.247816 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.248887 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.249612 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249681 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.249737 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.250165 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.250207 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260620 22:51:56.250440 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.246264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.2.90.131:43597)
0620 22:51:56.246434 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.246439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.246752 (+ 313us) server_negotiation.cc:407] Connection header received
0620 22:51:56.246812 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.246814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.246828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.246891 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:56.246974 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.247732 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249233 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.249879 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249956 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.250005 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.250329 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.250371 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260620 22:51:56.250757 3447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.245964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:42439 (local address 127.0.0.1:37026)
0620 22:51:56.246278 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.246317 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.246378 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.246592 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.246593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.246595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:56.246661 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.246662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.247992 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.247992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.249138 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.249141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.249804 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.249805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.249874 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.249878 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.249931 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.249974 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260620 22:51:56.250978 3239 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1f63a4fb264021aa45defc96775453 (DEFAULT_TABLE table=test-table [id=e6d7e7046c9f4ea889b8ecb3286140aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:56.251036 3373 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1f63a4fb264021aa45defc96775453 (DEFAULT_TABLE table=test-table [id=e6d7e7046c9f4ea889b8ecb3286140aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:56.251322 3239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1f63a4fb264021aa45defc96775453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:56.251351 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1f63a4fb264021aa45defc96775453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:56.251384 3105 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1f63a4fb264021aa45defc96775453 (DEFAULT_TABLE table=test-table [id=e6d7e7046c9f4ea889b8ecb3286140aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:56.251732 3105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1f63a4fb264021aa45defc96775453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:56.254490 3453 tablet_bootstrap.cc:492] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: Bootstrap starting.
I20260620 22:51:56.254716 3454 tablet_bootstrap.cc:492] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b: Bootstrap starting.
I20260620 22:51:56.254958 3455 tablet_bootstrap.cc:492] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9: Bootstrap starting.
I20260620 22:51:56.255095 3453 tablet_bootstrap.cc:654] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:56.255357 3454 tablet_bootstrap.cc:654] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:56.255585 3455 tablet_bootstrap.cc:654] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:56.255664 3453 log.cc:824] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:56.255901 3454 log.cc:824] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:56.256073 3455 log.cc:824] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:56.256558 3454 tablet_bootstrap.cc:492] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b: No bootstrap required, opened a new log
I20260620 22:51:56.256615 3454 ts_tablet_manager.cc:1397] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:51:56.256747 3453 tablet_bootstrap.cc:492] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: No bootstrap required, opened a new log
I20260620 22:51:56.256747 3455 tablet_bootstrap.cc:492] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9: No bootstrap required, opened a new log
I20260620 22:51:56.256810 3453 ts_tablet_manager.cc:1397] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:51:56.256810 3455 ts_tablet_manager.cc:1397] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:51:56.257936 3454 raft_consensus.cc:348] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.258064 3454 raft_consensus.cc:374] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:56.258095 3454 raft_consensus.cc:729] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec7c47a190d44ffab6fee9c902d9c5b, State: Initialized, Role: FOLLOWER
I20260620 22:51:56.258222 3454 consensus_queue.cc:260] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.258797 3454 ts_tablet_manager.cc:1428] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:51:56.258898 3455 raft_consensus.cc:348] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.259065 3455 raft_consensus.cc:374] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:56.259099 3455 raft_consensus.cc:729] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1798ab5d07b44d65970968f4a431e3c9, State: Initialized, Role: FOLLOWER
I20260620 22:51:56.259133 3453 raft_consensus.cc:348] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.259238 3455 consensus_queue.cc:260] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.259284 3453 raft_consensus.cc:374] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:56.259325 3453 raft_consensus.cc:729] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095090a6b8f345a7b66972e4c7ab9a12, State: Initialized, Role: FOLLOWER
I20260620 22:51:56.259467 3455 ts_tablet_manager.cc:1428] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:51:56.259441 3453 consensus_queue.cc:260] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.259578 3171 heartbeater.cc:503] Master 127.2.90.190:40787 was elected leader, sending a full tablet report...
I20260620 22:51:56.259675 3453 ts_tablet_manager.cc:1428] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:51:56.259684 3305 heartbeater.cc:503] Master 127.2.90.190:40787 was elected leader, sending a full tablet report...
I20260620 22:51:56.259790 3439 heartbeater.cc:503] Master 127.2.90.190:40787 was elected leader, sending a full tablet report...
I20260620 22:51:56.299706 3461 raft_consensus.cc:482] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:51:56.299841 3461 raft_consensus.cc:504] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.300187 3461 leader_election.cc:288] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1798ab5d07b44d65970968f4a431e3c9 (127.2.90.129:42439), 9ec7c47a190d44ffab6fee9c902d9c5b (127.2.90.131:43597)
I20260620 22:51:56.302857 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.300337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:43597 (local address 127.2.90.130:55773)
0620 22:51:56.300387 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.300398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.300439 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.300580 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.300582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.300584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:56.300646 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.300648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.301330 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.301331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.302099 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.302102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.302628 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.302628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.302696 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.302700 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.302748 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.302788 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:51:56.302867 3452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.300429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:55773 (local address 127.2.90.131:43597)
0620 22:51:56.300469 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.300472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.300480 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:51:56.300503 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.300504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.300514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.300566 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:56.300735 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.301262 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.302184 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.302561 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.302611 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.302618 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.302762 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.302800 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:51:56.302934 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.300396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:39819 (local address 127.2.90.129:42439)
0620 22:51:56.300454 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.300456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.300696 (+ 240us) server_negotiation.cc:407] Connection header received
0620 22:51:56.300728 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.300729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.300741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.300786 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:56.300905 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.301380 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.302246 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.302631 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.302696 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.302704 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.302848 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.302881 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:51:56.302958 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.300337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:42439 (local address 127.2.90.130:39819)
0620 22:51:56.300618 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:56.300626 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:56.300670 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:56.300790 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:56.300791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:56.300792 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:56.300852 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.300854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.301452 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.301452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.302168 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:56.302170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.302700 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.302701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:56.302778 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.302781 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:51:56.302831 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:51:56.302867 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":146,"thread_start_us":76,"threads_started":1}
I20260620 22:51:56.303164 3393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1f63a4fb264021aa45defc96775453" candidate_uuid: "095090a6b8f345a7b66972e4c7ab9a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" is_pre_election: true
I20260620 22:51:56.303267 3125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1f63a4fb264021aa45defc96775453" candidate_uuid: "095090a6b8f345a7b66972e4c7ab9a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1798ab5d07b44d65970968f4a431e3c9" is_pre_election: true
I20260620 22:51:56.303316 3393 raft_consensus.cc:2454] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 095090a6b8f345a7b66972e4c7ab9a12 in term 0.
I20260620 22:51:56.303419 3125 raft_consensus.cc:2454] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 095090a6b8f345a7b66972e4c7ab9a12 in term 0.
I20260620 22:51:56.303517 3194 leader_election.cc:302] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095090a6b8f345a7b66972e4c7ab9a12, 9ec7c47a190d44ffab6fee9c902d9c5b; no voters:
I20260620 22:51:56.303643 3461 raft_consensus.cc:2781] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:51:56.303682 3461 raft_consensus.cc:482] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:51:56.303702 3461 raft_consensus.cc:3037] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:56.304344 3461 raft_consensus.cc:504] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.304520 3461 leader_election.cc:288] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [CANDIDATE]: Term 1 election: Requested vote from peers 1798ab5d07b44d65970968f4a431e3c9 (127.2.90.129:42439), 9ec7c47a190d44ffab6fee9c902d9c5b (127.2.90.131:43597)
I20260620 22:51:56.304580 3125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1f63a4fb264021aa45defc96775453" candidate_uuid: "095090a6b8f345a7b66972e4c7ab9a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1798ab5d07b44d65970968f4a431e3c9"
I20260620 22:51:56.304633 3393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1f63a4fb264021aa45defc96775453" candidate_uuid: "095090a6b8f345a7b66972e4c7ab9a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b"
I20260620 22:51:56.304654 3125 raft_consensus.cc:3037] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:56.304719 3393 raft_consensus.cc:3037] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:56.305326 3125 raft_consensus.cc:2454] T 3e1f63a4fb264021aa45defc96775453 P 1798ab5d07b44d65970968f4a431e3c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 095090a6b8f345a7b66972e4c7ab9a12 in term 1.
I20260620 22:51:56.305464 3393 raft_consensus.cc:2454] T 3e1f63a4fb264021aa45defc96775453 P 9ec7c47a190d44ffab6fee9c902d9c5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 095090a6b8f345a7b66972e4c7ab9a12 in term 1.
I20260620 22:51:56.305493 3192 leader_election.cc:302] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095090a6b8f345a7b66972e4c7ab9a12, 1798ab5d07b44d65970968f4a431e3c9; no voters:
I20260620 22:51:56.305567 3461 raft_consensus.cc:2781] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:56.305686 3461 raft_consensus.cc:686] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [term 1 LEADER]: Becoming Leader. State: Replica: 095090a6b8f345a7b66972e4c7ab9a12, State: Running, Role: LEADER
I20260620 22:51:56.305763 3461 consensus_queue.cc:237] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } }
I20260620 22:51:56.306504 2979 catalog_manager.cc:5495] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 095090a6b8f345a7b66972e4c7ab9a12 (127.2.90.130). New cstate: current_term: 1 leader_uuid: "095090a6b8f345a7b66972e4c7ab9a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1798ab5d07b44d65970968f4a431e3c9" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 42439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "095090a6b8f345a7b66972e4c7ab9a12" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ec7c47a190d44ffab6fee9c902d9c5b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43597 } health_report { overall_health: UNKNOWN } } }
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
W20260620 22:51:56.325006 3306 tablet.cc:2369] T 3e1f63a4fb264021aa45defc96775453 P 095090a6b8f345a7b66972e4c7ab9a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:51:56.326897 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.320283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.2.90.129:42439)
0620 22:51:56.320346 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.320348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.320422 (+ 74us) server_negotiation.cc:407] Connection header received
0620 22:51:56.320489 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.320490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.320499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.320537 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:56.321051 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.321524 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.322182 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.322379 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.322431 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.324500 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:56.324508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:56.324508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:56.324514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:56.325982 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.326257 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.326259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.326260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.326309 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.326441 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.326442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.326442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.326597 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:56.326695 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.326754 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.326821 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:51:56.347934 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.341777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.2.90.129:42439)
0620 22:51:56.341840 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.341842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.341893 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:51:56.341966 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.341967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.341975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.342017 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:56.342563 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.343026 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.343726 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.343927 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.343984 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.345961 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:56.345962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:56.345963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:56.345967 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:56.347145 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.347394 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.347395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.347395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.347439 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.347561 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.347561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.347561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.347636 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:56.347697 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.347805 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.347857 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[2935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:51:56.368510 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:56.362447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.2.90.129:42439)
0620 22:51:56.362505 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:56.362507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:56.362598 (+ 91us) server_negotiation.cc:407] Connection header received
0620 22:51:56.362660 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:56.362661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:56.362671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:56.362715 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:56.363197 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.363657 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.364341 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:56.364524 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:56.364574 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:56.366677 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:56.366679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:56.366679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:56.366685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:56.367734 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.368009 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.368010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.368010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.368052 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:56.368168 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:56.368168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:56.368169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:56.368243 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:56.368305 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:56.368386 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:51:56.368446 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 22:51:56.368929 3085 tablet_service.cc:2908] Rejecting scan request for tablet 3e1f63a4fb264021aa45defc96775453: Uninitialized: clean time has not yet been initialized
I20260620 22:51:56.369616 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3042
I20260620 22:51:56.373596 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3176
I20260620 22:51:56.377413 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3310
I20260620 22:51:56.380844 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 2948
2026-06-20T22:51:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1789 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[3477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[3477](info): set up 2 sockets
Jun 20 22:51:56 dist-test-slave-4mtw krb5kdc[3477](info): commencing operation
krb5kdc: starting...
W20260620 22:51:57.327986 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:51:57.433516 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45183
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:45183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:57.509423 3490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:57.509618 3490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:57.509639 3490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:57.510759 3490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:51:57.510797 3490 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:51:57.510809 3490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:57.510833 3490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:57.510849 3490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:51:57.510871 3490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:51:57.512318 3490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:45183
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:57.512533 3490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:57.512744 3490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:57.517059 3496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:57.517060 3495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.517086 3490 server_base.cc:1034] running on GCE node
W20260620 22:51:57.517055 3498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.517520 3490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:57.517778 3490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:57.518913 3490 hybrid_clock.cc:648] HybridClock initialized: now 1781995917518902 us; error 24 us; skew 500 ppm
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:57.524547 3490 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:51:57.524971 3490 webserver.cc:460] Webserver started at http://127.2.90.190:36863/ using document root <none> and password file <none>
I20260620 22:51:57.525172 3490 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:57.525218 3490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:57.525345 3490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:57.526317 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "a4ca88c2393748d283dc98063439edc4"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "7dedd97df1ace992c3e24dfc74907a2c"
server_key_iv: "64a1cbb84e81a2976b8384fe2f4b6390"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.526626 3490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "a4ca88c2393748d283dc98063439edc4"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "7dedd97df1ace992c3e24dfc74907a2c"
server_key_iv: "64a1cbb84e81a2976b8384fe2f4b6390"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.527753 3490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:51:57.528345 3505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.528483 3490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:51:57.528551 3490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "a4ca88c2393748d283dc98063439edc4"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "7dedd97df1ace992c3e24dfc74907a2c"
server_key_iv: "64a1cbb84e81a2976b8384fe2f4b6390"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.528615 3490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:57.547717 3490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:57.549365 3490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:57.549490 3490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:57.553685 3490 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:45183
I20260620 22:51:57.553738 3567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:45183 every 8 connection(s)
I20260620 22:51:57.554044 3490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:51:57.554634 3568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:57.556879 3568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: Bootstrap starting.
I20260620 22:51:57.557156 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3490
I20260620 22:51:57.557231 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:51:57.557376 2410 external_mini_cluster.cc:1428] Setting key 57c7f357db86c3b8e9c867d65eba5006
I20260620 22:51:57.557493 3568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:57.557763 3568 log.cc:824] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:57.558372 3568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: No bootstrap required, opened a new log
I20260620 22:51:57.559676 3568 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ca88c2393748d283dc98063439edc4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45183 } }
I20260620 22:51:57.559808 3568 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:57.559828 3568 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ca88c2393748d283dc98063439edc4, State: Initialized, Role: FOLLOWER
I20260620 22:51:57.559945 3568 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ca88c2393748d283dc98063439edc4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45183 } }
I20260620 22:51:57.560009 3568 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:51:57.560032 3568 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:51:57.560076 3568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:57.560587 3568 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ca88c2393748d283dc98063439edc4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45183 } }
I20260620 22:51:57.560701 3568 leader_election.cc:302] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ca88c2393748d283dc98063439edc4; no voters:
I20260620 22:51:57.560868 3568 leader_election.cc:288] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:51:57.560911 3573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:57.561049 3573 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [term 1 LEADER]: Becoming Leader. State: Replica: a4ca88c2393748d283dc98063439edc4, State: Running, Role: LEADER
I20260620 22:51:57.561120 3568 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:51:57.561134 3573 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ca88c2393748d283dc98063439edc4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45183 } }
I20260620 22:51:57.561455 3573 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ca88c2393748d283dc98063439edc4. Latest consensus state: current_term: 1 leader_uuid: "a4ca88c2393748d283dc98063439edc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ca88c2393748d283dc98063439edc4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45183 } } }
I20260620 22:51:57.561540 3573 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:57.561463 3574 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ca88c2393748d283dc98063439edc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ca88c2393748d283dc98063439edc4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45183 } } }
I20260620 22:51:57.561672 3574 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4 [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:57.561728 3579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:51:57.562194 3579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:51:57.563685 3579 catalog_manager.cc:1347] Generated new cluster ID: 5ac09f1496b04715a87bf237f44bf8d4
I20260620 22:51:57.563736 3579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:57.567745 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.557823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.2.90.190:45183)
0620 22:51:57.557989 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:57.557998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:57.558029 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:51:57.558436 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:57.558445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:57.558503 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:57.558624 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:57.559212 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.560596 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.561688 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.562320 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.562418 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.564873 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:57.564877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:57.564877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:57.564888 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:57.566367 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.566699 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.566703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.566703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.566782 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.566957 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.566958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.566959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.567184 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:57.567334 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:57.567421 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:51:57.567550 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260620 22:51:57.573975 3579 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:51:57.574481 3579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:51:57.578382 3579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: Generated new TSK 0
I20260620 22:51:57.578544 3579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:51:57.604568 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45183
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:57.692766 3594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:57.692947 3594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:57.692970 3594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:57.694097 3594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:57.694135 3594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:57.694154 3594 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:51:57.694170 3594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:51:57.695551 3594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:45183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:57.695756 3594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:57.695967 3594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:57.700567 3599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:57.700548 3600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:57.700537 3602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.700750 3594 server_base.cc:1034] running on GCE node
I20260620 22:51:57.701040 3594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:57.701256 3594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:57.702425 3594 hybrid_clock.cc:648] HybridClock initialized: now 1781995917702412 us; error 36 us; skew 500 ppm
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:57.704615 3594 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:51:57.705003 3594 webserver.cc:460] Webserver started at http://127.2.90.129:35509/ using document root <none> and password file <none>
I20260620 22:51:57.705184 3594 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:57.705221 3594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:57.705307 3594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:57.706243 3594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "f60e92474db841dabffbacc2de0e9434"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "7a13f437d342116717271234bbca22f2"
server_key_iv: "779432ad22ec21de75af131b6af0bbcb"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.706528 3594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "f60e92474db841dabffbacc2de0e9434"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "7a13f437d342116717271234bbca22f2"
server_key_iv: "779432ad22ec21de75af131b6af0bbcb"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.707612 3594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:51:57.708177 3609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.708330 3594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.708418 3594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "f60e92474db841dabffbacc2de0e9434"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "7a13f437d342116717271234bbca22f2"
server_key_iv: "779432ad22ec21de75af131b6af0bbcb"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.708472 3594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:57.726293 3594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:57.728195 3594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:57.728327 3594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:57.728866 3594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:57.728919 3594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.728950 3594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:57.728965 3594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.734963 3594 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37099
I20260620 22:51:57.735116 3724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37099 every 8 connection(s)
I20260620 22:51:57.735378 3594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:57.738821 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.729356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:60119 (local address 127.2.90.190:45183)
0620 22:51:57.729418 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:57.729421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:57.729687 (+ 266us) server_negotiation.cc:407] Connection header received
0620 22:51:57.730246 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:57.730248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:57.730261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:57.730340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:57.731137 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.731902 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.733528 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.733842 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.733929 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.736357 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:57.736359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:57.736359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:57.736367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:57.737612 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.737976 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.737978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.737978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.738031 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.738240 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.738241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.738241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.738351 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:57.738403 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:57.738653 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 22:51:57.738732 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:51:57.738976 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.729332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.2.90.129:60119)
0620 22:51:57.729585 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:57.729614 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:57.730132 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:57.730425 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:57.730430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:57.730764 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:57.731031 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.731037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.732002 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.732003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.733343 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.733345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.733924 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.733926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.734066 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.734621 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:57.734629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:57.736272 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:57.737698 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.737704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.737706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.737912 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.738083 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.738084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.738084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.738187 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.738402 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:57.738415 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:57.738542 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:51:57.738626 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:51:57.738773 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260620 22:51:57.739260 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3594
I20260620 22:51:57.739338 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:51:57.739501 2410 external_mini_cluster.cc:1428] Setting key 5039de1df9683b4d3d0d381e91e008d8
I20260620 22:51:57.739491 3725 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45183
I20260620 22:51:57.739573 3725 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:57.739770 3725 heartbeater.cc:511] Master 127.2.90.190:45183 requested a full tablet report, sending...
I20260620 22:51:57.740288 3521 ts_manager.cc:194] Registered new tserver with Master: f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099)
I20260620 22:51:57.741135 3521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:60119
I20260620 22:51:57.746130 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.740854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.2.90.129:43047)
0620 22:51:57.740962 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:57.740970 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:57.741014 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:57.741124 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:57.741125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:57.741319 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:57.741378 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.741380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.742145 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.742146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.742878 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.742880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.743201 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.743202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.743278 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.743650 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:57.743653 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:57.743899 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:57.745272 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.745272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.745273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.745430 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.745588 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.745588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.745588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.745642 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.745840 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:57.745840 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:57.745884 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:51:57.745943 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:51:57.746040 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260620 22:51:57.746125 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.740905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:43047 (local address 127.2.90.190:45183)
0620 22:51:57.740939 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:57.740941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:57.741007 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:51:57.741056 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:57.741056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:57.741065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:57.741122 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:57.741437 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.742062 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.742942 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.743145 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.743199 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.743971 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:57.743972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:57.743972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:57.743978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:57.745202 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.745489 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.745490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.745490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.745538 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.745689 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.745690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.745691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.745787 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:57.745842 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:57.745971 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:51:57.746037 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:51:57.761374 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45183
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:57.838290 3734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:57.838474 3734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:57.838495 3734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:57.839587 3734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:57.839623 3734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:57.839641 3734 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:51:57.839665 3734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:51:57.840992 3734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:45183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:57.841150 3734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:57.841311 3734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:57.845587 3739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:57.845592 3742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.845618 3734 server_base.cc:1034] running on GCE node
W20260620 22:51:57.845592 3740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.845894 3734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:57.846096 3734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:57.847220 3734 hybrid_clock.cc:648] HybridClock initialized: now 1781995917847209 us; error 26 us; skew 500 ppm
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:57.849409 3734 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:51:57.849776 3734 webserver.cc:460] Webserver started at http://127.2.90.130:35769/ using document root <none> and password file <none>
I20260620 22:51:57.849938 3734 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:57.849973 3734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:57.850055 3734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:57.850948 3734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "724cc3aa644841da88f8b3904e102507"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "b5373c1a6556dcae784f0318e34a497d"
server_key_iv: "1d70133a02def65d3c91e788bf40a352"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.851213 3734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "724cc3aa644841da88f8b3904e102507"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "b5373c1a6556dcae784f0318e34a497d"
server_key_iv: "1d70133a02def65d3c91e788bf40a352"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.852299 3734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:51:57.852928 3749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.853065 3734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.853137 3734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "724cc3aa644841da88f8b3904e102507"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "b5373c1a6556dcae784f0318e34a497d"
server_key_iv: "1d70133a02def65d3c91e788bf40a352"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.853188 3734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:57.869455 3734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:57.871094 3734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:57.871198 3734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:57.871613 3734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:57.871651 3734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.871672 3734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:57.871683 3734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:57.877605 3734 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:37125
I20260620 22:51:57.877686 3864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:37125 every 8 connection(s)
I20260620 22:51:57.877934 3734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:57.880761 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.872060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:48537 (local address 127.2.90.190:45183)
0620 22:51:57.872111 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:57.872113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:57.872337 (+ 224us) server_negotiation.cc:407] Connection header received
0620 22:51:57.872870 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:57.872871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:57.872884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:57.872951 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:57.873727 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.874465 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.875879 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.876160 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.876234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.878434 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:57.878435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:57.878436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:57.878443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:57.879606 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.879945 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.879946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.879947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.879994 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.880195 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.880196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.880196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.880294 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:57.880348 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:57.880584 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 22:51:57.880652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:51:57.880923 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.872063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.2.90.130:48537)
0620 22:51:57.872209 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:57.872256 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:57.872777 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:57.873024 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:57.873029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:57.873343 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:57.873621 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.873627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.874579 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.874580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.875792 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.875794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.876239 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.876239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.876333 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.876836 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:57.876846 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:57.878352 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:57.879679 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.879684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.879687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.879884 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.880045 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.880046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.880046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.880143 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.880347 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:57.880353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:57.880476 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 22:51:57.880558 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:51:57.880723 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260620 22:51:57.881353 3865 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45183
I20260620 22:51:57.881421 3865 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:57.881567 3865 heartbeater.cc:511] Master 127.2.90.190:45183 requested a full tablet report, sending...
I20260620 22:51:57.881882 3522 ts_manager.cc:194] Registered new tserver with Master: 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125)
I20260620 22:51:57.882370 3522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:48537
I20260620 22:51:57.885177 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3734
I20260620 22:51:57.885250 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:51:57.885422 2410 external_mini_cluster.cc:1428] Setting key 9f1d16304f7cf68452652932c9606357
I20260620 22:51:57.886982 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.882292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:51109 (local address 127.2.90.190:45183)
0620 22:51:57.882364 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:57.882366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:57.882374 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:51:57.882430 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:57.882431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:57.882439 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:57.882492 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:57.882813 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.883250 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.883882 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.884049 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.884088 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.884790 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:57.884791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:57.884792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:57.884797 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:57.886039 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.886317 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.886318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.886319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.886368 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:57.886516 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:57.886517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:57.886517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:57.886648 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:57.886702 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:57.886831 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:51:57.886896 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 22:51:57.886996 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:57.882212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.2.90.130:51109)
0620 22:51:57.882326 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:57.882336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:57.882380 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:57.882511 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:57.882512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:57.882685 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:57.882746 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.882748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.883340 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.883341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.883829 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:57.883830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:57.884093 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:57.884094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:57.884139 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:57.884429 (+ 290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:57.884432 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:57.884711 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:57.886108 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.886110 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.886110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.886256 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.886417 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:57.886418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:57.886418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:57.886470 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:57.886701 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:57.886702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:57.886743 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:51:57.886809 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:51:57.886898 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:51:57.907086 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45183
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:57.984665 3874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:57.984906 3874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:57.984928 3874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:57.986152 3874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:57.986194 3874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:57.986214 3874 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:51:57.986229 3874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:51:57.987635 3874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:45183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:57.987854 3874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:57.988058 3874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:57.992667 3879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.992767 3874 server_base.cc:1034] running on GCE node
W20260620 22:51:57.992702 3882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:57.992726 3880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:57.993079 3874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:57.993284 3874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:57.994418 3874 hybrid_clock.cc:648] HybridClock initialized: now 1781995917994397 us; error 31 us; skew 500 ppm
Jun 20 22:51:57 dist-test-slave-4mtw krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:57.996641 3874 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:51:57.997040 3874 webserver.cc:460] Webserver started at http://127.2.90.131:33879/ using document root <none> and password file <none>
I20260620 22:51:57.997210 3874 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:57.997246 3874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:57.997330 3874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:57.998299 3874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "66f1523d2d62496fb35ead5914497ee3"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "1d93b77020b617d575915e493c8d3d7d"
server_key_iv: "919408f2c2ed738d6ea2b18aa0fdaf88"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.998591 3874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "66f1523d2d62496fb35ead5914497ee3"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "1d93b77020b617d575915e493c8d3d7d"
server_key_iv: "919408f2c2ed738d6ea2b18aa0fdaf88"
server_key_version: "encryptionkey@0"
I20260620 22:51:57.999763 3874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:51:58.000380 3889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.000517 3874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:51:58.000594 3874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "66f1523d2d62496fb35ead5914497ee3"
format_stamp: "Formatted at 2026-06-20 22:51:57 on dist-test-slave-4mtw"
server_key: "1d93b77020b617d575915e493c8d3d7d"
server_key_iv: "919408f2c2ed738d6ea2b18aa0fdaf88"
server_key_version: "encryptionkey@0"
I20260620 22:51:58.000661 3874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:58.010510 3874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:58.012513 3874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:58.012614 3874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:58.013079 3874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:58.013118 3874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.013139 3874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:58.013149 3874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.018873 3874 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:35223
I20260620 22:51:58.019001 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:35223 every 8 connection(s)
I20260620 22:51:58.019220 3874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:58.020977 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 3874
I20260620 22:51:58.021070 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:51:58.021238 2410 external_mini_cluster.cc:1428] Setting key 37b99d5a0a9c3dff5fbb746316a71757
I20260620 22:51:58.022678 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.013486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:47365 (local address 127.2.90.190:45183)
0620 22:51:58.013549 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.013551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.013768 (+ 217us) server_negotiation.cc:407] Connection header received
0620 22:51:58.014316 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.014317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.014329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.014409 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:58.015188 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.015940 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.017382 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.017677 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.017761 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.020150 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:58.020152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:58.020152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:58.020159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:58.021471 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.021889 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.021889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.021890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.021948 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.022230 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.022230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.022230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.022329 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:58.022375 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.022559 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.022617 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:51:58.022835 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.013453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.2.90.131:47365)
0620 22:51:58.013669 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.013699 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.014197 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.014491 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.014495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.014816 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:58.015095 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.015101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.016030 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.016031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.017298 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.017301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.017767 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.017768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.017905 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.018432 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:58.018441 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:58.020071 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:58.021552 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:58.021557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:58.021560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:58.021825 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:58.022072 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:58.022073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:58.022073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:58.022182 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:58.022367 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:58.022370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:58.022469 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.022540 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.022647 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":64,"threads_started":1}
I20260620 22:51:58.023221 4005 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45183
I20260620 22:51:58.023290 4005 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:58.023455 4005 heartbeater.cc:511] Master 127.2.90.190:45183 requested a full tablet report, sending...
I20260620 22:51:58.023758 3521 ts_manager.cc:194] Registered new tserver with Master: 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223)
I20260620 22:51:58.024109 3521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:47365
I20260620 22:51:58.028530 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.024176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:43125 (local address 127.2.90.190:45183)
0620 22:51:58.024209 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.024211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.024323 (+ 112us) server_negotiation.cc:407] Connection header received
0620 22:51:58.024374 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.024374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.024384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.024435 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:58.024716 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.025209 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.025821 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.025988 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.026028 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.026670 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:58.026671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:58.026671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:58.026676 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:58.027790 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.028028 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.028029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.028029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.028068 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.028171 (+ 103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.028171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.028171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.028247 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:58.028286 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.028431 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.028477 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:51:58.028558 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.024119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.2.90.131:43125)
0620 22:51:58.024257 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.024267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.024323 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.024438 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.024438 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.024582 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:58.024626 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.024628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.025278 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.025279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.025760 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.025761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.026032 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.026033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.026080 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.026369 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:58.026372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:58.026610 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:58.027851 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:58.027852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:58.027852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:58.027983 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:58.028101 (+ 118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:58.028101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:58.028101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:58.028139 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:58.028305 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:58.028305 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:58.028343 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.028416 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.028492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
I20260620 22:51:58.032866 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:58.064544 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.057820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.2.90.190:45183)
0620 22:51:58.057954 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.057956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.057968 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:51:58.058001 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.058002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.058015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.058100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:58.058701 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.059442 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.060147 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.060326 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.060367 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.062489 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:58.062491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:58.062491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:58.062499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:58.063748 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.064024 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.064025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.064025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.064070 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.064194 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.064194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.064194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.064285 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:58.064352 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.064421 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.064474 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 22:51:58.066093 3521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:51:58.067572 3521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:51:58.073349 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.069475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37099 (local address 127.0.0.1:54918)
0620 22:51:58.069991 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.069999 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.070046 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.070200 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.070200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.070202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.070241 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.070242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.071487 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.071488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.072232 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.072234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.073054 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.073055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.073128 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.073141 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.073215 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.073279 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":3,"thread_start_us":52,"threads_started":1}
I20260620 22:51:58.073374 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.069630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.2.90.129:37099)
0620 22:51:58.069779 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.069784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.070062 (+ 278us) server_negotiation.cc:407] Connection header received
0620 22:51:58.070106 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.070108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.070123 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.070197 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.070323 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.071396 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.072315 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.072972 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.073041 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.073098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.073235 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.073272 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260620 22:51:58.073529 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.069475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37125 (local address 127.0.0.1:45364)
0620 22:51:58.069847 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.069858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.069917 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.070131 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.070132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.070134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.070188 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.070190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.071464 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.071465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.072501 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.072503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.073340 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.073341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.073395 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.073398 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.073444 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.073475 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":51,"threads_started":1}
I20260620 22:51:58.073750 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.069684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.2.90.130:37125)
0620 22:51:58.069855 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.069860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.069904 (+ 44us) server_negotiation.cc:407] Connection header received
0620 22:51:58.069978 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.069980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.069994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.070061 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.070259 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.071380 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.072583 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.073205 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.073259 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.073310 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.073611 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.073650 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260620 22:51:58.073932 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.069472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35223 (local address 127.0.0.1:52774)
0620 22:51:58.069782 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.069824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.069900 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.070100 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.070103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.070104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.070165 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.070167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.071164 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.071165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.072586 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.072587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.073725 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.073726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.073791 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.073794 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.073824 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.073859 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":47,"threads_started":1}
I20260620 22:51:58.074239 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.069696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.2.90.131:35223)
0620 22:51:58.069858 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.069861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.069870 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:51:58.069987 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.069989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.070004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.070236 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.070359 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.071087 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.072653 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.073380 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.074029 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.074093 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.074116 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.074150 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260620 22:51:58.074123 3659 tablet_service.cc:1467] Processing CreateTablet for tablet 8073922bb2f64bb8abce7d5ccc50e9d0 (DEFAULT_TABLE table=test-table [id=d898a14204df4a5288007f731a8c84a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:58.074316 3799 tablet_service.cc:1467] Processing CreateTablet for tablet 8073922bb2f64bb8abce7d5ccc50e9d0 (DEFAULT_TABLE table=test-table [id=d898a14204df4a5288007f731a8c84a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:58.074503 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8073922bb2f64bb8abce7d5ccc50e9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.074599 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8073922bb2f64bb8abce7d5ccc50e9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.077251 3939 tablet_service.cc:1467] Processing CreateTablet for tablet 8073922bb2f64bb8abce7d5ccc50e9d0 (DEFAULT_TABLE table=test-table [id=d898a14204df4a5288007f731a8c84a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:58.077564 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8073922bb2f64bb8abce7d5ccc50e9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.078069 4025 tablet_bootstrap.cc:492] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Bootstrap starting.
I20260620 22:51:58.078675 4025 tablet_bootstrap.cc:654] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.079011 4025 log.cc:824] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:58.079334 4026 tablet_bootstrap.cc:492] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Bootstrap starting.
I20260620 22:51:58.079666 4025 tablet_bootstrap.cc:492] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: No bootstrap required, opened a new log
I20260620 22:51:58.079708 4025 ts_tablet_manager.cc:1397] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:51:58.079897 4026 tablet_bootstrap.cc:654] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.080190 4026 log.cc:824] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:58.080495 4028 tablet_bootstrap.cc:492] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Bootstrap starting.
I20260620 22:51:58.081434 4028 tablet_bootstrap.cc:654] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.081331 4025 raft_consensus.cc:348] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.081480 4025 raft_consensus.cc:374] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.081524 4025 raft_consensus.cc:729] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724cc3aa644841da88f8b3904e102507, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.081646 4025 consensus_queue.cc:260] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.081717 4028 log.cc:824] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:58.081900 4025 ts_tablet_manager.cc:1428] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:51:58.081918 3865 heartbeater.cc:503] Master 127.2.90.190:45183 was elected leader, sending a full tablet report...
I20260620 22:51:58.082366 4026 tablet_bootstrap.cc:492] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: No bootstrap required, opened a new log
I20260620 22:51:58.082432 4026 ts_tablet_manager.cc:1397] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:51:58.082837 4028 tablet_bootstrap.cc:492] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: No bootstrap required, opened a new log
I20260620 22:51:58.082901 4028 ts_tablet_manager.cc:1397] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:51:58.084292 4026 raft_consensus.cc:348] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.084450 4026 raft_consensus.cc:374] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.084491 4026 raft_consensus.cc:729] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60e92474db841dabffbacc2de0e9434, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.084599 4026 consensus_queue.cc:260] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.084733 4028 raft_consensus.cc:348] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.084877 4028 raft_consensus.cc:374] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.084882 4026 ts_tablet_manager.cc:1428] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:51:58.084908 4028 raft_consensus.cc:729] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f1523d2d62496fb35ead5914497ee3, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.084913 3725 heartbeater.cc:503] Master 127.2.90.190:45183 was elected leader, sending a full tablet report...
I20260620 22:51:58.085001 4028 consensus_queue.cc:260] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.085233 4028 ts_tablet_manager.cc:1428] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 22:51:58.085352 4005 heartbeater.cc:503] Master 127.2.90.190:45183 was elected leader, sending a full tablet report...
W20260620 22:51:58.128428 3866 tablet.cc:2369] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:51:58.145255 4032 raft_consensus.cc:482] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:51:58.145360 4032 raft_consensus.cc:504] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.145660 4032 leader_election.cc:288] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223), 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125)
I20260620 22:51:58.148280 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.145887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:41659 (local address 127.2.90.131:35223)
0620 22:51:58.145942 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.145944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.146002 (+ 58us) server_negotiation.cc:407] Connection header received
0620 22:51:58.146074 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.146075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.146086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.146133 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.146276 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.146735 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.147631 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.147987 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.148046 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.148055 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.148183 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.148222 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:51:58.148280 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.145811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35223 (local address 127.2.90.129:41659)
0620 22:51:58.145952 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.145964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.146015 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.146139 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.146140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.146141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.146208 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.146209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.146804 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.146805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.147552 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.147554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.148059 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.148060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.148124 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.148127 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.148170 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.148205 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 22:51:58.148586 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8073922bb2f64bb8abce7d5ccc50e9d0" candidate_uuid: "f60e92474db841dabffbacc2de0e9434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1523d2d62496fb35ead5914497ee3" is_pre_election: true
I20260620 22:51:58.148773 3959 raft_consensus.cc:2454] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f60e92474db841dabffbacc2de0e9434 in term 0.
I20260620 22:51:58.148861 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.145834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37125 (local address 127.2.90.129:45175)
0620 22:51:58.146012 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.146021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.146079 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.146200 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.146200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.146202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.146690 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.146693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.147281 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.147282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.147996 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.147998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.148609 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.148611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.148699 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.148703 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.148752 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.148786 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260620 22:51:58.148888 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.145898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:45175 (local address 127.2.90.130:37125)
0620 22:51:58.145942 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.145944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.146086 (+ 142us) server_negotiation.cc:407] Connection header received
0620 22:51:58.146137 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.146138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.146149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.146198 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.146752 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.147216 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.148075 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.148525 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.148586 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.148594 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.148769 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.148820 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:51:58.149062 3611 leader_election.cc:302] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1523d2d62496fb35ead5914497ee3, f60e92474db841dabffbacc2de0e9434; no voters:
I20260620 22:51:58.149137 3819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8073922bb2f64bb8abce7d5ccc50e9d0" candidate_uuid: "f60e92474db841dabffbacc2de0e9434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724cc3aa644841da88f8b3904e102507" is_pre_election: true
I20260620 22:51:58.149181 4032 raft_consensus.cc:2781] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:51:58.149235 4032 raft_consensus.cc:482] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:51:58.149266 4032 raft_consensus.cc:3037] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.149281 3819 raft_consensus.cc:2454] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f60e92474db841dabffbacc2de0e9434 in term 0.
I20260620 22:51:58.150151 4032 raft_consensus.cc:504] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.150260 4032 leader_election.cc:288] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [CANDIDATE]: Term 1 election: Requested vote from peers 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223), 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125)
I20260620 22:51:58.150430 3819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8073922bb2f64bb8abce7d5ccc50e9d0" candidate_uuid: "f60e92474db841dabffbacc2de0e9434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724cc3aa644841da88f8b3904e102507"
I20260620 22:51:58.150437 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8073922bb2f64bb8abce7d5ccc50e9d0" candidate_uuid: "f60e92474db841dabffbacc2de0e9434" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1523d2d62496fb35ead5914497ee3"
I20260620 22:51:58.150493 3819 raft_consensus.cc:3037] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.150508 3959 raft_consensus.cc:3037] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.151166 3819 raft_consensus.cc:2454] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f60e92474db841dabffbacc2de0e9434 in term 1.
I20260620 22:51:58.151206 3959 raft_consensus.cc:2454] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f60e92474db841dabffbacc2de0e9434 in term 1.
I20260620 22:51:58.151324 3610 leader_election.cc:302] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724cc3aa644841da88f8b3904e102507, f60e92474db841dabffbacc2de0e9434; no voters:
I20260620 22:51:58.151417 4032 raft_consensus.cc:2781] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:58.151558 4032 raft_consensus.cc:686] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 1 LEADER]: Becoming Leader. State: Replica: f60e92474db841dabffbacc2de0e9434, State: Running, Role: LEADER
I20260620 22:51:58.151634 4032 consensus_queue.cc:237] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } }
I20260620 22:51:58.152398 3522 catalog_manager.cc:5495] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 reported cstate change: term changed from 0 to 1, leader changed from <none> to f60e92474db841dabffbacc2de0e9434 (127.2.90.129). New cstate: current_term: 1 leader_uuid: "f60e92474db841dabffbacc2de0e9434" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } health_report { overall_health: UNKNOWN } } }
I20260620 22:51:58.164198 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.161882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.2.90.129:37099)
0620 22:51:58.161927 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.161928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.161973 (+ 45us) server_negotiation.cc:407] Connection header received
0620 22:51:58.162096 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.162097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.162115 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.162170 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:51:58.162287 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.162738 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.163487 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.163670 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.163721 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.163901 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.163994 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.164061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.164113 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.164132 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:51:58.165730 3959 raft_consensus.cc:1261] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Refusing update from remote peer f60e92474db841dabffbacc2de0e9434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:51:58.165745 3819 raft_consensus.cc:1261] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Refusing update from remote peer f60e92474db841dabffbacc2de0e9434: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:51:58.165995 4032 consensus_queue.cc:1035] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:51:58.166110 4035 consensus_queue.cc:1035] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:51:58.168366 4039 mvcc.cc:205] Tried to move back new op lower bound from 7299055280804556800 to 7299055280749596672. Current Snapshot: MvccSnapshot[applied={T|T < 7299055280804556800}]
I20260620 22:51:58.168764 4040 mvcc.cc:205] Tried to move back new op lower bound from 7299055280804556800 to 7299055280749596672. Current Snapshot: MvccSnapshot[applied={T|T < 7299055280804556800 or (T in {7299055280804556800})}]
I20260620 22:51:58.171406 3521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:51:58.171482 3521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:51:58.172099 3521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: Sending DeleteTablet for 3 replicas of tablet 8073922bb2f64bb8abce7d5ccc50e9d0
I20260620 22:51:58.172379 3939 tablet_service.cc:1514] Processing DeleteTablet for tablet 8073922bb2f64bb8abce7d5ccc50e9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:51:58 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:52774
I20260620 22:51:58.172384 3799 tablet_service.cc:1514] Processing DeleteTablet for tablet 8073922bb2f64bb8abce7d5ccc50e9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:51:58 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:45364
I20260620 22:51:58.172461 3659 tablet_service.cc:1514] Processing DeleteTablet for tablet 8073922bb2f64bb8abce7d5ccc50e9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:51:58 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:54918
I20260620 22:51:58.172737 4049 tablet_replica.cc:330] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: stopping tablet replica
I20260620 22:51:58.172835 4049 raft_consensus.cc:2229] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:51:58.172804 3521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:51:58.172933 3521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:51:58.172935 4049 raft_consensus.cc:2258] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:51:58.173275 4048 tablet_replica.cc:330] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: stopping tablet replica
I20260620 22:51:58.173344 4049 ts_tablet_manager.cc:1905] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:51:58.173458 4048 raft_consensus.cc:2229] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:51:58.173596 4048 raft_consensus.cc:2258] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:51:58.173774 4050 tablet_replica.cc:330] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: stopping tablet replica
I20260620 22:51:58.173902 4050 raft_consensus.cc:2229] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:51:58.174043 4050 raft_consensus.cc:2258] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:51:58.174152 3799 tablet_service.cc:1467] Processing CreateTablet for tablet a03bff0a22c94b89881999568a72fa14 (DEFAULT_TABLE table=test-table [id=361e3600b6da4b33a32a202914850d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:58.174268 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03bff0a22c94b89881999568a72fa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.175181 4049 ts_tablet_manager.cc:1918] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:51:58.175246 4049 log.cc:1196] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/8073922bb2f64bb8abce7d5ccc50e9d0
I20260620 22:51:58.175478 4049 ts_tablet_manager.cc:1939] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 66f1523d2d62496fb35ead5914497ee3: Deleting consensus metadata
I20260620 22:51:58.175537 4025 tablet_bootstrap.cc:492] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Bootstrap starting.
I20260620 22:51:58.175580 3939 tablet_service.cc:1467] Processing CreateTablet for tablet a03bff0a22c94b89881999568a72fa14 (DEFAULT_TABLE table=test-table [id=361e3600b6da4b33a32a202914850d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:58.175652 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03bff0a22c94b89881999568a72fa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.175818 3507 catalog_manager.cc:4841] TS 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223): tablet 8073922bb2f64bb8abce7d5ccc50e9d0 (table test-table [id=d898a14204df4a5288007f731a8c84a6]) successfully deleted
I20260620 22:51:58.175927 4025 tablet_bootstrap.cc:654] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.176559 4028 tablet_bootstrap.cc:492] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Bootstrap starting.
I20260620 22:51:58.176672 4025 tablet_bootstrap.cc:492] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: No bootstrap required, opened a new log
I20260620 22:51:58.176738 4025 ts_tablet_manager.cc:1397] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:51:58.176872 4025 raft_consensus.cc:348] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.176926 4025 raft_consensus.cc:374] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.176945 4025 raft_consensus.cc:729] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724cc3aa644841da88f8b3904e102507, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.176963 4028 tablet_bootstrap.cc:654] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.176985 4025 consensus_queue.cc:260] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.177017 3659 tablet_service.cc:1467] Processing CreateTablet for tablet a03bff0a22c94b89881999568a72fa14 (DEFAULT_TABLE table=test-table [id=361e3600b6da4b33a32a202914850d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:51:58.177060 4025 ts_tablet_manager.cc:1428] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.177109 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03bff0a22c94b89881999568a72fa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.178090 4026 tablet_bootstrap.cc:492] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Bootstrap starting.
I20260620 22:51:58.178434 4026 tablet_bootstrap.cc:654] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.178431 4048 ts_tablet_manager.cc:1905] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:51:58.178583 4028 tablet_bootstrap.cc:492] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: No bootstrap required, opened a new log
I20260620 22:51:58.178622 4028 ts_tablet_manager.cc:1397] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:51:58.178742 4028 raft_consensus.cc:348] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.178807 4028 raft_consensus.cc:374] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.178825 4028 raft_consensus.cc:729] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f1523d2d62496fb35ead5914497ee3, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.178865 4028 consensus_queue.cc:260] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.178946 4028 ts_tablet_manager.cc:1428] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.179188 4026 tablet_bootstrap.cc:492] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: No bootstrap required, opened a new log
I20260620 22:51:58.179230 4026 ts_tablet_manager.cc:1397] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:51:58.179345 4026 raft_consensus.cc:348] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.179407 4026 raft_consensus.cc:374] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.179425 4026 raft_consensus.cc:729] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60e92474db841dabffbacc2de0e9434, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.179469 4026 consensus_queue.cc:260] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.179543 4026 ts_tablet_manager.cc:1428] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.179777 4048 ts_tablet_manager.cc:1918] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:51:58.179844 4048 log.cc:1196] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/8073922bb2f64bb8abce7d5ccc50e9d0
I20260620 22:51:58.180086 4048 ts_tablet_manager.cc:1939] T 8073922bb2f64bb8abce7d5ccc50e9d0 P 724cc3aa644841da88f8b3904e102507: Deleting consensus metadata
I20260620 22:51:58.180414 3506 catalog_manager.cc:4841] TS 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125): tablet 8073922bb2f64bb8abce7d5ccc50e9d0 (table test-table [id=d898a14204df4a5288007f731a8c84a6]) successfully deleted
I20260620 22:51:58.180931 4050 ts_tablet_manager.cc:1905] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:51:58.182076 4050 ts_tablet_manager.cc:1918] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:51:58.182129 4050 log.cc:1196] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/8073922bb2f64bb8abce7d5ccc50e9d0
I20260620 22:51:58.182307 4050 ts_tablet_manager.cc:1939] T 8073922bb2f64bb8abce7d5ccc50e9d0 P f60e92474db841dabffbacc2de0e9434: Deleting consensus metadata
I20260620 22:51:58.182659 3508 catalog_manager.cc:4841] TS f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099): tablet 8073922bb2f64bb8abce7d5ccc50e9d0 (table test-table [id=d898a14204df4a5288007f731a8c84a6]) successfully deleted
I20260620 22:51:58.209685 4030 raft_consensus.cc:482] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:51:58.209760 4030 raft_consensus.cc:504] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.210040 4030 leader_election.cc:288] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099), 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223)
I20260620 22:51:58.213519 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.210259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:45555 (local address 127.2.90.131:35223)
0620 22:51:58.210304 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.210305 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.210628 (+ 323us) server_negotiation.cc:407] Connection header received
0620 22:51:58.210653 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.210654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.210674 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.210720 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.210946 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.211565 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.212602 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.213161 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.213231 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.213240 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.213404 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.213448 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:51:58.213613 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.210320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:43717 (local address 127.2.90.129:37099)
0620 22:51:58.210365 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.210367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.210375 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:51:58.210399 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.210400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.210430 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.210492 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.210619 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.211268 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.212742 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.213271 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.213338 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.213347 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.213516 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.213550 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:51:58.213627 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.210180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37099 (local address 127.2.90.130:43717)
0620 22:51:58.210238 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.210250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.210285 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.210497 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.210498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.210500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.210564 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.210565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.211356 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.211357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.212628 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.212631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.213376 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.213377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.213460 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.213464 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.213511 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.213548 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:51:58.213867 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03bff0a22c94b89881999568a72fa14" candidate_uuid: "724cc3aa644841da88f8b3904e102507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60e92474db841dabffbacc2de0e9434" is_pre_election: true
I20260620 22:51:58.213994 3679 raft_consensus.cc:2454] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724cc3aa644841da88f8b3904e102507 in term 0.
I20260620 22:51:58.214079 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.210182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35223 (local address 127.2.90.130:45555)
0620 22:51:58.210422 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.210431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.210479 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.210745 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.210745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.210747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.210835 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.210837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.211634 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.211635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.212528 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.212530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.213238 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.213239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.213311 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.213316 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.213952 (+ 636us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.213992 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":143,"thread_start_us":59,"threads_started":1}
I20260620 22:51:58.214178 3752 leader_election.cc:302] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724cc3aa644841da88f8b3904e102507, f60e92474db841dabffbacc2de0e9434; no voters:
I20260620 22:51:58.214295 4030 raft_consensus.cc:2781] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:51:58.214334 4030 raft_consensus.cc:482] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:51:58.214354 4030 raft_consensus.cc:3037] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.214432 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03bff0a22c94b89881999568a72fa14" candidate_uuid: "724cc3aa644841da88f8b3904e102507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1523d2d62496fb35ead5914497ee3" is_pre_election: true
I20260620 22:51:58.214511 3959 raft_consensus.cc:2454] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724cc3aa644841da88f8b3904e102507 in term 0.
I20260620 22:51:58.215050 4030 raft_consensus.cc:504] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.215158 4030 leader_election.cc:288] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [CANDIDATE]: Term 1 election: Requested vote from peers f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099), 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223)
I20260620 22:51:58.215319 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03bff0a22c94b89881999568a72fa14" candidate_uuid: "724cc3aa644841da88f8b3904e102507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60e92474db841dabffbacc2de0e9434"
I20260620 22:51:58.215317 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03bff0a22c94b89881999568a72fa14" candidate_uuid: "724cc3aa644841da88f8b3904e102507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f1523d2d62496fb35ead5914497ee3"
I20260620 22:51:58.215385 3959 raft_consensus.cc:3037] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.215385 3679 raft_consensus.cc:3037] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.216089 3959 raft_consensus.cc:2454] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724cc3aa644841da88f8b3904e102507 in term 1.
I20260620 22:51:58.216121 3679 raft_consensus.cc:2454] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724cc3aa644841da88f8b3904e102507 in term 1.
I20260620 22:51:58.216255 3751 leader_election.cc:302] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1523d2d62496fb35ead5914497ee3, 724cc3aa644841da88f8b3904e102507; no voters:
I20260620 22:51:58.216336 4030 raft_consensus.cc:2781] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:58.216455 4030 raft_consensus.cc:686] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 1 LEADER]: Becoming Leader. State: Replica: 724cc3aa644841da88f8b3904e102507, State: Running, Role: LEADER
I20260620 22:51:58.216519 4030 consensus_queue.cc:237] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.217170 3521 catalog_manager.cc:5495] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 reported cstate change: term changed from 0 to 1, leader changed from <none> to 724cc3aa644841da88f8b3904e102507 (127.2.90.130). New cstate: current_term: 1 leader_uuid: "724cc3aa644841da88f8b3904e102507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } health_report { overall_health: UNKNOWN } } }
W20260620 22:51:58.236048 3726 tablet.cc:2369] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995917, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:58.256263 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.249276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45183 (local address 127.0.0.1:38932)
0620 22:51:58.249423 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.249435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.249516 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.249652 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.249652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.250063 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:58.250121 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.250124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.250787 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.250788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.251642 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.251645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.251952 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.251953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.252015 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.252517 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:58.252522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:58.253984 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:58.255317 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:58.255323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:58.255325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:58.255533 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:58.255689 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:58.255690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:58.255690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:58.255760 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:58.255950 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:58.255957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:58.256000 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.256079 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.256179 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260620 22:51:58.256353 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.249350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.2.90.190:45183)
0620 22:51:58.249397 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.249400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.249504 (+ 104us) server_negotiation.cc:407] Connection header received
0620 22:51:58.249572 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.249573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.249581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.249630 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:58.250206 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.250701 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.251716 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.251885 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.251928 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.254058 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:58.254059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:58.254060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:58.254067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:58.255253 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.255592 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.255594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.255594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.255640 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.255805 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.255806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.255806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.255898 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:58.255953 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.256129 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.256246 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":26}
I20260620 22:51:58.258183 3521 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:38932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:51:58.258373 3521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:51:58.259709 3799 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2ed41b67244928a1baef80adc44a23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23e34e4b033f40559bd4593b04016901]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:51:58.259709 3939 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2ed41b67244928a1baef80adc44a23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23e34e4b033f40559bd4593b04016901]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:51:58.259853 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2ed41b67244928a1baef80adc44a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.259852 3939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2ed41b67244928a1baef80adc44a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.261283 4025 tablet_bootstrap.cc:492] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: Bootstrap starting.
I20260620 22:51:58.261339 3659 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2ed41b67244928a1baef80adc44a23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23e34e4b033f40559bd4593b04016901]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:51:58.261451 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2ed41b67244928a1baef80adc44a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:58.261684 4025 tablet_bootstrap.cc:654] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.262307 4025 tablet_bootstrap.cc:492] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: No bootstrap required, opened a new log
I20260620 22:51:58.262349 4025 ts_tablet_manager.cc:1397] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:51:58.262537 4026 tablet_bootstrap.cc:492] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: Bootstrap starting.
I20260620 22:51:58.262578 4028 tablet_bootstrap.cc:492] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: Bootstrap starting.
I20260620 22:51:58.262610 4025 raft_consensus.cc:348] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.262673 4025 raft_consensus.cc:374] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.262692 4025 raft_consensus.cc:729] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724cc3aa644841da88f8b3904e102507, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.262734 4025 consensus_queue.cc:260] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.262825 4025 ts_tablet_manager.cc:1428] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.263104 4028 tablet_bootstrap.cc:654] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.263180 4026 tablet_bootstrap.cc:654] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:58.263862 4030 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.263939 4026 tablet_bootstrap.cc:492] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: No bootstrap required, opened a new log
I20260620 22:51:58.263939 4030 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:51:58.263981 4026 ts_tablet_manager.cc:1397] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:51:58.264214 4026 raft_consensus.cc:348] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.264277 4026 raft_consensus.cc:374] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.264295 4026 raft_consensus.cc:729] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60e92474db841dabffbacc2de0e9434, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.264338 4026 consensus_queue.cc:260] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.264417 4026 ts_tablet_manager.cc:1428] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:51:58.264490 4032 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.264515 4028 tablet_bootstrap.cc:492] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: No bootstrap required, opened a new log
I20260620 22:51:58.264544 4032 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:51:58.264556 4028 ts_tablet_manager.cc:1397] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:51:58.264664 4028 raft_consensus.cc:348] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.264744 4028 raft_consensus.cc:374] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:58.264765 4028 raft_consensus.cc:729] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f1523d2d62496fb35ead5914497ee3, State: Initialized, Role: FOLLOWER
I20260620 22:51:58.264801 4028 consensus_queue.cc:260] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.264878 4028 ts_tablet_manager.cc:1428] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:58.265084 4033 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.265147 4033 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 22:51:58.269711 4006 tablet.cc:2369] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:51:58.451592 4033 raft_consensus.cc:482] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:51:58.451686 4033 raft_consensus.cc:504] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.452010 4033 leader_election.cc:288] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099), 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125)
I20260620 22:51:58.454885 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.452272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:60315 (local address 127.2.90.129:37099)
0620 22:51:58.452342 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.452344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.452352 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:51:58.452378 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.452379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.452388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.452443 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.452600 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.453134 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.454078 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.454598 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.454672 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.454682 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.454802 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.454831 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:51:58.454922 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.452185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37099 (local address 127.2.90.131:60315)
0620 22:51:58.452240 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.452252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.452296 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.452461 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.452462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.452464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.452534 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.452537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.453199 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.453200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.453990 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.453993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.454671 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.454671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.454743 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.454747 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.454794 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.454830 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":6}
I20260620 22:51:58.455156 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2ed41b67244928a1baef80adc44a23" candidate_uuid: "66f1523d2d62496fb35ead5914497ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60e92474db841dabffbacc2de0e9434" is_pre_election: true
I20260620 22:51:58.455261 3679 raft_consensus.cc:2454] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f1523d2d62496fb35ead5914497ee3 in term 0.
I20260620 22:51:58.455488 3892 leader_election.cc:302] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1523d2d62496fb35ead5914497ee3, f60e92474db841dabffbacc2de0e9434; no voters:
I20260620 22:51:58.455516 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.452325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:43831 (local address 127.2.90.130:37125)
0620 22:51:58.452366 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.452368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.452466 (+ 98us) server_negotiation.cc:407] Connection header received
0620 22:51:58.452497 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.452498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.452508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.452563 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:51:58.452748 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.453207 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.454678 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.455169 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.455229 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.455236 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.455420 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.455466 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:51:58.455610 4033 raft_consensus.cc:2781] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:51:58.455648 4033 raft_consensus.cc:482] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:51:58.455682 4033 raft_consensus.cc:3037] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.455765 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.452206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37125 (local address 127.2.90.131:43831)
0620 22:51:58.452404 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.452412 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.452459 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.452578 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.452579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.452580 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:51:58.452652 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.452654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.453278 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.453279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.454598 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.454600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.455246 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.455247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.455328 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.455332 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.455589 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.455635 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
I20260620 22:51:58.456071 3819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2ed41b67244928a1baef80adc44a23" candidate_uuid: "66f1523d2d62496fb35ead5914497ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724cc3aa644841da88f8b3904e102507" is_pre_election: true
I20260620 22:51:58.456146 3819 raft_consensus.cc:2454] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f1523d2d62496fb35ead5914497ee3 in term 0.
I20260620 22:51:58.456523 4033 raft_consensus.cc:504] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.456655 4033 leader_election.cc:288] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [CANDIDATE]: Term 1 election: Requested vote from peers f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099), 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125)
I20260620 22:51:58.456799 3819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2ed41b67244928a1baef80adc44a23" candidate_uuid: "66f1523d2d62496fb35ead5914497ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724cc3aa644841da88f8b3904e102507"
I20260620 22:51:58.456810 3679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2ed41b67244928a1baef80adc44a23" candidate_uuid: "66f1523d2d62496fb35ead5914497ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60e92474db841dabffbacc2de0e9434"
I20260620 22:51:58.456866 3819 raft_consensus.cc:3037] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.456871 3679 raft_consensus.cc:3037] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:58.457374 3819 raft_consensus.cc:2454] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f1523d2d62496fb35ead5914497ee3 in term 1.
I20260620 22:51:58.457485 3679 raft_consensus.cc:2454] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f1523d2d62496fb35ead5914497ee3 in term 1.
I20260620 22:51:58.457535 3890 leader_election.cc:302] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f1523d2d62496fb35ead5914497ee3, 724cc3aa644841da88f8b3904e102507; no voters:
I20260620 22:51:58.457616 4033 raft_consensus.cc:2781] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:58.457748 4033 raft_consensus.cc:686] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 66f1523d2d62496fb35ead5914497ee3, State: Running, Role: LEADER
I20260620 22:51:58.457823 4033 consensus_queue.cc:237] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } }
I20260620 22:51:58.458145 4067 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: TxnStatusTablet state changed. Reason: New leader 66f1523d2d62496fb35ead5914497ee3. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.458227 4067 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:51:58.458451 4069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:51:58.458544 4069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:51:58.458504 3522 catalog_manager.cc:5495] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131). New cstate: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } health_report { overall_health: HEALTHY } } }
I20260620 22:51:58.489923 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.487238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35223 (local address 127.0.0.1:52782)
0620 22:51:58.487341 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.487353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.487408 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.487562 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.487563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.487564 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:51:58.487607 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.487609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.488455 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.488455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.489066 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.489068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.489502 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.489503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.489546 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.489559 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.489806 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.489810 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.489830 (+ 20us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.489862 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 22:51:58.489940 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.487301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.2.90.131:35223)
0620 22:51:58.487356 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.487358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.487391 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:51:58.487470 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.487471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.487490 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.487559 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:51:58.487645 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.488391 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.489146 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.489438 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.489500 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.489634 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.489754 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.489823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.489844 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.489862 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:51:58.490937 3819 raft_consensus.cc:1261] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Refusing update from remote peer 66f1523d2d62496fb35ead5914497ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:51:58.491011 3679 raft_consensus.cc:1261] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Refusing update from remote peer 66f1523d2d62496fb35ead5914497ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:51:58.491164 4067 consensus_queue.cc:1035] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:51:58.491298 4033 consensus_queue.cc:1035] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:51:58.491926 4032 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: TxnStatusTablet state changed. Reason: New leader 66f1523d2d62496fb35ead5914497ee3. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.492013 4032 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:51:58.491981 4030 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: TxnStatusTablet state changed. Reason: New leader 66f1523d2d62496fb35ead5914497ee3. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.492050 4030 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:51:58.492882 4033 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.492954 4033 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:51:58.493005 4030 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.493084 4030 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 724cc3aa644841da88f8b3904e102507: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:51:58.493636 4074 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.493774 4074 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P 66f1523d2d62496fb35ead5914497ee3: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:51:58.494117 4032 tablet_replica.cc:439] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66f1523d2d62496fb35ead5914497ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 } } peers { permanent_uuid: "724cc3aa644841da88f8b3904e102507" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37125 } } peers { permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 } } }
I20260620 22:51:58.494194 4032 tablet_replica.cc:442] T 7a2ed41b67244928a1baef80adc44a23 P f60e92474db841dabffbacc2de0e9434: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:51:58.496555 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.494543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.2.90.130:37125)
0620 22:51:58.494598 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.494600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.494606 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:51:58.494638 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.494639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.494656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.494711 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:51:58.494800 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.495241 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.495896 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.496089 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.496137 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.496267 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.496364 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.496418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.496469 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.496484 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:51:58.501076 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.498549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:44211 (local address 127.2.90.131:35223)
0620 22:51:58.498606 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.498608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.498703 (+ 95us) server_negotiation.cc:407] Connection header received
0620 22:51:58.498738 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.498739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.498748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.498809 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:51:58.498884 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.499576 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.500270 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.500568 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.500631 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.500813 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.500860 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.500903 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.500993 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.501009 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:51:58.501268 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.498507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35223 (local address 127.2.90.130:44211)
0620 22:51:58.498642 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.498651 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.498703 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.498812 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.498812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.498813 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:51:58.498849 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.498851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.499639 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.499639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.500201 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.500203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.500640 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.500641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.500721 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.500735 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.500929 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.500932 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.501146 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.501186 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260620 22:51:58.503736 3959 raft_consensus.cc:1261] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Refusing update from remote peer 724cc3aa644841da88f8b3904e102507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:51:58.503755 3679 raft_consensus.cc:1261] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Refusing update from remote peer 724cc3aa644841da88f8b3904e102507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:51:58.503957 4055 consensus_queue.cc:1035] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f60e92474db841dabffbacc2de0e9434" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:51:58.504081 4030 consensus_queue.cc:1035] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f1523d2d62496fb35ead5914497ee3" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:51:58.511951 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.509489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:58649 (local address 127.2.90.130:37125)
0620 22:51:58.509524 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.509526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.509553 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:51:58.509604 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.509605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.509611 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.509651 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:51:58.509789 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.510218 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.511329 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.511524 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.511573 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.511743 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.511774 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.511817 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.511868 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.511884 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:51:58.511967 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.509447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37125 (local address 127.2.90.131:58649)
0620 22:51:58.509504 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:58.509516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:58.509567 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:58.509672 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:58.509673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:58.509675 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:51:58.509732 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.509734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.510276 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.510277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.511234 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:58.511237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.511583 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.511583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:58.511668 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.511680 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:51:58.511826 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:51:58.511827 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:51:58.511866 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 22:51:58.511906 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 22:51:58.521145 3520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:51:58.521257 3520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:51:58.521885 3520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4ca88c2393748d283dc98063439edc4: Sending DeleteTablet for 3 replicas of tablet a03bff0a22c94b89881999568a72fa14
I20260620 22:51:58.522178 3659 tablet_service.cc:1514] Processing DeleteTablet for tablet a03bff0a22c94b89881999568a72fa14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:51:58 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:54918
I20260620 22:51:58.522210 3939 tablet_service.cc:1514] Processing DeleteTablet for tablet a03bff0a22c94b89881999568a72fa14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:51:58 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:52774
I20260620 22:51:58.522280 4050 tablet_replica.cc:330] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: stopping tablet replica
I20260620 22:51:58.522306 4049 tablet_replica.cc:330] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: stopping tablet replica
I20260620 22:51:58.522333 4050 raft_consensus.cc:2229] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:51:58.522349 4049 raft_consensus.cc:2229] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:51:58.522377 4050 raft_consensus.cc:2258] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:51:58.522380 4049 raft_consensus.cc:2258] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:51:58.522797 4049 ts_tablet_manager.cc:1905] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:51:58.522782 4050 ts_tablet_manager.cc:1905] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:51:58.524528 4049 ts_tablet_manager.cc:1918] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:51:58.524578 4049 log.cc:1196] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/a03bff0a22c94b89881999568a72fa14
I20260620 22:51:58.524825 4049 ts_tablet_manager.cc:1939] T a03bff0a22c94b89881999568a72fa14 P 66f1523d2d62496fb35ead5914497ee3: Deleting consensus metadata
I20260620 22:51:58.524900 3799 tablet_service.cc:1514] Processing DeleteTablet for tablet a03bff0a22c94b89881999568a72fa14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:51:58 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:45364
I20260620 22:51:58.524991 4048 tablet_replica.cc:330] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: stopping tablet replica
I20260620 22:51:58.525059 4048 raft_consensus.cc:2229] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:51:58.525143 4048 raft_consensus.cc:2258] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:51:58.525242 3507 catalog_manager.cc:4841] TS 66f1523d2d62496fb35ead5914497ee3 (127.2.90.131:35223): tablet a03bff0a22c94b89881999568a72fa14 (table test-table [id=361e3600b6da4b33a32a202914850d83]) successfully deleted
I20260620 22:51:58.525328 4050 ts_tablet_manager.cc:1918] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:51:58.525375 4050 log.cc:1196] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/a03bff0a22c94b89881999568a72fa14
I20260620 22:51:58.525463 4048 ts_tablet_manager.cc:1905] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:51:58.525557 4050 ts_tablet_manager.cc:1939] T a03bff0a22c94b89881999568a72fa14 P f60e92474db841dabffbacc2de0e9434: Deleting consensus metadata
I20260620 22:51:58.525835 3508 catalog_manager.cc:4841] TS f60e92474db841dabffbacc2de0e9434 (127.2.90.129:37099): tablet a03bff0a22c94b89881999568a72fa14 (table test-table [id=361e3600b6da4b33a32a202914850d83]) successfully deleted
I20260620 22:51:58.526901 4048 ts_tablet_manager.cc:1918] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:51:58.526971 4048 log.cc:1196] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/a03bff0a22c94b89881999568a72fa14
I20260620 22:51:58.527184 4048 ts_tablet_manager.cc:1939] T a03bff0a22c94b89881999568a72fa14 P 724cc3aa644841da88f8b3904e102507: Deleting consensus metadata
I20260620 22:51:58.527477 3506 catalog_manager.cc:4841] TS 724cc3aa644841da88f8b3904e102507 (127.2.90.130:37125): tablet a03bff0a22c94b89881999568a72fa14 (table test-table [id=361e3600b6da4b33a32a202914850d83]) successfully deleted
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[3477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:51:58.538178 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.525474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.2.90.129:37099)
0620 22:51:58.525514 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.525516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.528861 (+ 3345us) server_negotiation.cc:407] Connection header received
0620 22:51:58.528914 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.528915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.528926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.528987 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:58.529632 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.530358 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.531397 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.531726 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.531814 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.535086 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:58.535095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:58.535095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:58.535103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:58.536978 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.537367 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.537370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.537370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.537431 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.537627 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.537628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.537628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.537779 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:58.537922 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.538003 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.538081 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:51:58.538653 3719 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54938
I20260620 22:51:58.547255 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:58.539543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.2.90.190:45183)
0620 22:51:58.539607 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:58.539609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:58.540355 (+ 746us) server_negotiation.cc:407] Connection header received
0620 22:51:58.540383 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:58.540384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:58.540412 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:58.540465 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:58.541106 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.541847 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.542823 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:58.543110 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:58.543180 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:58.544447 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:58.544449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:58.544449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:58.544456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:58.546148 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.546504 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.546506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.546506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.546569 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:58.546754 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:58.546755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:58.546755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:58.546885 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:58.546983 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:58.547095 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 22:51:58.547169 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 22:51:58.547452 3520 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38938
I20260620 22:51:58.548043 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3594
I20260620 22:51:58.557503 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3734
I20260620 22:51:58.562361 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3874
I20260620 22:51:58.567651 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 3490
2026-06-20T22:51:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2186 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[4090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[4090](info): set up 2 sockets
Jun 20 22:51:58 dist-test-slave-4mtw krb5kdc[4090](info): commencing operation
krb5kdc: starting...
W20260620 22:51:59.528075 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:51:59 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:51:59.633541 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37577
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:37577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:59.710848 4103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:59.711051 4103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:59.711073 4103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:59.712203 4103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:51:59.712260 4103 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:51:59.712273 4103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:59.712287 4103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:59.712311 4103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:51:59.712327 4103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:51:59.713795 4103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:37577
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:59.714015 4103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:59.714221 4103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:51:59.718633 4103 server_base.cc:1034] running on GCE node
W20260620 22:51:59.718603 4108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:59.718587 4111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:59.718600 4109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:59.719060 4103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:59.719338 4103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:59.720472 4103 hybrid_clock.cc:648] HybridClock initialized: now 1781995919720461 us; error 34 us; skew 500 ppm
Jun 20 22:51:59 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:59.722700 4103 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:51:59.723100 4103 webserver.cc:460] Webserver started at http://127.2.90.190:34921/ using document root <none> and password file <none>
I20260620 22:51:59.723306 4103 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:59.723352 4103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:59.723472 4103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:59.724452 4103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "7b528a4888794018ba1e574ac463822b"
format_stamp: "Formatted at 2026-06-20 22:51:59 on dist-test-slave-4mtw"
server_key: "90a6ac90fa17b7b3242f287eec2c717d"
server_key_iv: "c67419921cc0fe947f2dafd30cc92fdf"
server_key_version: "encryptionkey@0"
I20260620 22:51:59.724793 4103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "7b528a4888794018ba1e574ac463822b"
format_stamp: "Formatted at 2026-06-20 22:51:59 on dist-test-slave-4mtw"
server_key: "90a6ac90fa17b7b3242f287eec2c717d"
server_key_iv: "c67419921cc0fe947f2dafd30cc92fdf"
server_key_version: "encryptionkey@0"
I20260620 22:51:59.725931 4103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:51:59.726545 4118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:59.726693 4103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:51:59.726770 4103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "7b528a4888794018ba1e574ac463822b"
format_stamp: "Formatted at 2026-06-20 22:51:59 on dist-test-slave-4mtw"
server_key: "90a6ac90fa17b7b3242f287eec2c717d"
server_key_iv: "c67419921cc0fe947f2dafd30cc92fdf"
server_key_version: "encryptionkey@0"
I20260620 22:51:59.726840 4103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:59.755609 4103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:59.757436 4103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:59.757568 4103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:59.761641 4103 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:37577
I20260620 22:51:59.761708 4180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:37577 every 8 connection(s)
I20260620 22:51:59.761977 4103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:51:59.762580 4181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:51:59.764770 4181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Bootstrap starting.
I20260620 22:51:59.765307 4181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Neither blocks nor log segments found. Creating new log.
I20260620 22:51:59.765554 4181 log.cc:824] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:51:59.766155 4181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: No bootstrap required, opened a new log
I20260620 22:51:59.767792 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 4103
I20260620 22:51:59.767891 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:51:59.768055 2410 external_mini_cluster.cc:1428] Setting key ba8c86bad03d9d990e050254c6065b57
I20260620 22:51:59.768299 4181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b528a4888794018ba1e574ac463822b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37577 } }
I20260620 22:51:59.768431 4181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:51:59.768462 4181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b528a4888794018ba1e574ac463822b, State: Initialized, Role: FOLLOWER
I20260620 22:51:59.768597 4181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b528a4888794018ba1e574ac463822b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37577 } }
I20260620 22:51:59.768699 4181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:51:59.768743 4181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:51:59.768790 4181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:51:59.769264 4181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b528a4888794018ba1e574ac463822b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37577 } }
I20260620 22:51:59.769356 4181 leader_election.cc:302] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b528a4888794018ba1e574ac463822b; no voters:
I20260620 22:51:59.769493 4181 leader_election.cc:288] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:51:59.769536 4186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:51:59.769665 4186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [term 1 LEADER]: Becoming Leader. State: Replica: 7b528a4888794018ba1e574ac463822b, State: Running, Role: LEADER
I20260620 22:51:59.769721 4181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:51:59.769747 4186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b528a4888794018ba1e574ac463822b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37577 } }
I20260620 22:51:59.770045 4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b528a4888794018ba1e574ac463822b. Latest consensus state: current_term: 1 leader_uuid: "7b528a4888794018ba1e574ac463822b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b528a4888794018ba1e574ac463822b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37577 } } }
I20260620 22:51:59.770170 4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:59.770061 4188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b528a4888794018ba1e574ac463822b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b528a4888794018ba1e574ac463822b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37577 } } }
I20260620 22:51:59.770309 4188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b [sys.catalog]: This master's current role is: LEADER
I20260620 22:51:59.770411 4193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:51:59.770820 4193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:51:59.772324 4193 catalog_manager.cc:1347] Generated new cluster ID: ec8813b894cf46cabfff08b66320405f
I20260620 22:51:59.772365 4193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:51:59 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:59.778723 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:59.768529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.2.90.190:37577)
0620 22:51:59.768706 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:59.768715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:59.768748 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:51:59.768871 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:59.768878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:59.768926 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:59.769039 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:59.769688 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.771065 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.771980 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.772600 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.772751 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:59.775327 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:59.775331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:59.775331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:59.775342 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:59.777116 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:59.777485 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:59.777487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:59.777488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:59.777591 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:59.777775 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:59.777776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:59.777776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:59.778044 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:59.778179 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:59.778290 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 22:51:59.778527 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260620 22:51:59.792821 4193 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:51:59.793529 4193 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:51:59.800289 4193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Generated new TSK 0
I20260620 22:51:59.800452 4193 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:51:59.826772 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37577
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:51:59.905107 4207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:51:59.905292 4207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:51:59.905323 4207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:51:59.906392 4207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:51:59.906430 4207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:51:59.906457 4207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:51:59.906472 4207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:51:59.906494 4207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:51:59.907862 4207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:51:59.908071 4207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:51:59.908252 4207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:51:59.912801 4212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:59.912809 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:51:59.912839 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:51:59.912899 4207 server_base.cc:1034] running on GCE node
I20260620 22:51:59.913067 4207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:51:59.913277 4207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:51:59.914412 4207 hybrid_clock.cc:648] HybridClock initialized: now 1781995919914394 us; error 34 us; skew 500 ppm
Jun 20 22:51:59 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:51:59.916623 4207 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:51:59.917019 4207 webserver.cc:460] Webserver started at http://127.2.90.129:41809/ using document root <none> and password file <none>
I20260620 22:51:59.917183 4207 fs_manager.cc:359] Metadata directory not provided
I20260620 22:51:59.917220 4207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:51:59.917306 4207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:51:59.918253 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "1d988799520a46e0ae52b2c10fe67b2c"
format_stamp: "Formatted at 2026-06-20 22:51:59 on dist-test-slave-4mtw"
server_key: "6f06e3529aae8c6903616c3122b028a8"
server_key_iv: "37dd7d38c74973819b44a2de1df22587"
server_key_version: "encryptionkey@0"
I20260620 22:51:59.918531 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "1d988799520a46e0ae52b2c10fe67b2c"
format_stamp: "Formatted at 2026-06-20 22:51:59 on dist-test-slave-4mtw"
server_key: "6f06e3529aae8c6903616c3122b028a8"
server_key_iv: "37dd7d38c74973819b44a2de1df22587"
server_key_version: "encryptionkey@0"
I20260620 22:51:59.919597 4207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:51:59.920173 4222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:59.920301 4207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:59.920379 4207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "1d988799520a46e0ae52b2c10fe67b2c"
format_stamp: "Formatted at 2026-06-20 22:51:59 on dist-test-slave-4mtw"
server_key: "6f06e3529aae8c6903616c3122b028a8"
server_key_iv: "37dd7d38c74973819b44a2de1df22587"
server_key_version: "encryptionkey@0"
I20260620 22:51:59.920434 4207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:51:59.934784 4207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:51:59.936803 4207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:51:59.936940 4207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:51:59.937402 4207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:51:59.937438 4207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:59.937465 4207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:51:59.937484 4207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:51:59.943377 4207 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:45385
I20260620 22:51:59.943439 4337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:45385 every 8 connection(s)
I20260620 22:51:59.943715 4207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:51:59 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:51:59.946480 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:59.937786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:59679 (local address 127.2.90.190:37577)
0620 22:51:59.937851 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:59.937853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:59.938112 (+ 259us) server_negotiation.cc:407] Connection header received
0620 22:51:59.938616 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:59.938617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:59.938635 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:59.938700 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:59.939488 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.940222 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.941631 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.941944 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.942019 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:59.944304 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:59.944305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:59.944305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:59.944314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:59.945389 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:59.945712 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:59.945713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:59.945713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:59.945764 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:59.945945 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:59.945945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:59.945946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:59.946038 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:59.946091 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:59.946353 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 22:51:59.946417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:51:59.946600 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:59.937765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.2.90.129:59679)
0620 22:51:59.938008 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:59.938036 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:59.938499 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:59.938788 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:59.938792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:59.939120 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:59.939385 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:59.939391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.940307 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.940308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:59.941555 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:59.941558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.942018 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.942019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:59.942141 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:59.942621 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:59.942631 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:59.944227 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:59.945448 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:59.945453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:59.945454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:59.945647 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:59.945821 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:59.945821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:59.945822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:59.945905 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:59.946112 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:59.946117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:59.946228 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 22:51:59.946325 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 22:51:59.946442 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260620 22:51:59.947067 4338 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37577
I20260620 22:51:59.947144 4338 heartbeater.cc:464] Registering TS with master...
I20260620 22:51:59.947324 4338 heartbeater.cc:511] Master 127.2.90.190:37577 requested a full tablet report, sending...
I20260620 22:51:59.947805 4134 ts_manager.cc:194] Registered new tserver with Master: 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385)
I20260620 22:51:59.948662 4134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:59679
I20260620 22:51:59.951166 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 4207
I20260620 22:51:59.951244 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:51:59.951423 2410 external_mini_cluster.cc:1428] Setting key 452cc978b084a643294b461b089a0282
I20260620 22:51:59.952986 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:59.948043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:50365 (local address 127.2.90.190:37577)
0620 22:51:59.948079 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:51:59.948081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:51:59.948176 (+ 95us) server_negotiation.cc:407] Connection header received
0620 22:51:59.948227 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:51:59.948228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:51:59.948235 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:51:59.948280 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:51:59.948572 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.949112 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.949792 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.949980 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.950031 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:59.950818 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:51:59.950819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:51:59.950820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:51:59.950825 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:51:59.952028 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:59.952322 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:59.952323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:59.952323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:59.952374 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:51:59.952524 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:51:59.952525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:51:59.952525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:51:59.952621 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:51:59.952672 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:51:59.952826 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 22:51:59.952898 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:51:59.952996 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:51:59.947992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.2.90.129:50365)
0620 22:51:59.948119 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:51:59.948128 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:51:59.948180 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:51:59.948283 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:51:59.948284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:51:59.948456 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:51:59.948516 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:59.948517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.949207 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.949208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:59.949724 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:51:59.949725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:51:59.950034 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:51:59.950035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:51:59.950104 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:51:59.950462 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:51:59.950467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:51:59.950740 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:51:59.952097 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:59.952098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:59.952098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:59.952261 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:59.952423 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:51:59.952424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:51:59.952424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:51:59.952478 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:51:59.952670 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:51:59.952671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:51:59.952729 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:51:59.952806 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:51:59.952899 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:51:59.972402 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37577
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:00.049523 4347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:00.049717 4347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:00.049749 4347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:00.050889 4347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:00.050927 4347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:00.050954 4347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:52:00.050968 4347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:52:00.050992 4347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:00.052436 4347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:00.052665 4347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:00.052884 4347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:00.057591 4352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:00.057642 4353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:00.057596 4355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:00.057780 4347 server_base.cc:1034] running on GCE node
I20260620 22:52:00.057971 4347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:00.058156 4347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:00.059298 4347 hybrid_clock.cc:648] HybridClock initialized: now 1781995920059289 us; error 29 us; skew 500 ppm
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:00.061614 4347 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:00.061975 4347 webserver.cc:460] Webserver started at http://127.2.90.130:40243/ using document root <none> and password file <none>
I20260620 22:52:00.062153 4347 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:00.062191 4347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:00.062276 4347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:00.063237 4347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "e31c02f3350a406ca0bd3b1eb81646df"
format_stamp: "Formatted at 2026-06-20 22:52:00 on dist-test-slave-4mtw"
server_key: "26990214b34d592877ce843da4c66303"
server_key_iv: "c4289393039259a80924b838617ec746"
server_key_version: "encryptionkey@0"
I20260620 22:52:00.063522 4347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "e31c02f3350a406ca0bd3b1eb81646df"
format_stamp: "Formatted at 2026-06-20 22:52:00 on dist-test-slave-4mtw"
server_key: "26990214b34d592877ce843da4c66303"
server_key_iv: "c4289393039259a80924b838617ec746"
server_key_version: "encryptionkey@0"
I20260620 22:52:00.064612 4347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:00.065270 4362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.065418 4347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.065492 4347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "e31c02f3350a406ca0bd3b1eb81646df"
format_stamp: "Formatted at 2026-06-20 22:52:00 on dist-test-slave-4mtw"
server_key: "26990214b34d592877ce843da4c66303"
server_key_iv: "c4289393039259a80924b838617ec746"
server_key_version: "encryptionkey@0"
I20260620 22:52:00.065542 4347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:00.072671 4347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:00.074218 4347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:00.074314 4347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:00.074704 4347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:00.074740 4347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.074759 4347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:00.074769 4347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.080360 4347 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:42849
I20260620 22:52:00.080473 4477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:42849 every 8 connection(s)
I20260620 22:52:00.080708 4347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:00.083550 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.075108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:44315 (local address 127.2.90.190:37577)
0620 22:52:00.075165 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.075167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.075391 (+ 224us) server_negotiation.cc:407] Connection header received
0620 22:52:00.075915 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.075916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.075930 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.076000 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:00.076757 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.077492 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.078845 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.079069 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.079121 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.081355 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:00.081356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:00.081356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:00.081363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:00.082602 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.082910 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.082911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.082911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.082955 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.083108 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.083108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.083108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.083197 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:00.083239 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.083430 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.083489 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:00.083671 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.075077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.2.90.130:44315)
0620 22:52:00.075289 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.075324 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.075744 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.076062 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.076066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.076384 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:00.076637 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.076642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.077575 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.077576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.078764 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.078766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.079132 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.079133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.079260 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.079752 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:00.079762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:00.081277 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:00.082663 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.082666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.082668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.082857 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.082990 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.082991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.082991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.083064 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.083238 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:00.083240 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:00.083337 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.083408 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.083511 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260620 22:52:00.084079 4478 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37577
I20260620 22:52:00.084154 4478 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:00.084307 4478 heartbeater.cc:511] Master 127.2.90.190:37577 requested a full tablet report, sending...
I20260620 22:52:00.084666 4135 ts_manager.cc:194] Registered new tserver with Master: e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849)
I20260620 22:52:00.085152 4135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:44315
I20260620 22:52:00.086046 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 4347
I20260620 22:52:00.086117 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:00.086262 2410 external_mini_cluster.cc:1428] Setting key 0cb3283e996773025de4ae178eec4929
I20260620 22:52:00.090018 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.085020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:43319 (local address 127.2.90.190:37577)
0620 22:52:00.085057 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.085059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.085123 (+ 64us) server_negotiation.cc:407] Connection header received
0620 22:52:00.085167 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.085168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.085178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.085229 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:00.085565 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.086141 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.086818 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.087013 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.087063 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.087798 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:00.087812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:00.087813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:00.087820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:00.089040 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.089353 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.089354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.089354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.089395 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.089556 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.089556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.089557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.089709 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:00.089765 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.089911 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.089962 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:00.090092 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.084950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.2.90.130:43319)
0620 22:52:00.085078 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.085087 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.085132 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.085268 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.085269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.085441 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:00.085499 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.085501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.086219 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.086220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.086753 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.086755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.087068 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.087069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.087139 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.087478 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:00.087482 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:00.087718 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:00.089106 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.089107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.089107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.089293 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.089445 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.089445 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.089446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.089506 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.089771 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:00.089772 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:00.089818 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.089893 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.089993 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:00.106921 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37577
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:00.183830 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:00.184019 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:00.184051 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:00.185186 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:00.185225 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:00.185251 4487 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 22:52:00.185268 4487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:52:00.185298 4487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:00.186700 4487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:37577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:00.186926 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:00.187119 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:00.191649 4495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:00.191627 4492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:00.191609 4493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:00.191676 4487 server_base.cc:1034] running on GCE node
I20260620 22:52:00.191946 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:00.192133 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:00.193262 4487 hybrid_clock.cc:648] HybridClock initialized: now 1781995920193254 us; error 34 us; skew 500 ppm
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:00.195444 4487 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:00.195815 4487 webserver.cc:460] Webserver started at http://127.2.90.131:34435/ using document root <none> and password file <none>
I20260620 22:52:00.196007 4487 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:00.196053 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:00.196170 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:00.197237 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "6339a4c09a734011876f3e0ff0b112cf"
format_stamp: "Formatted at 2026-06-20 22:52:00 on dist-test-slave-4mtw"
server_key: "7b41a339de59dea60d449a7761cde2a6"
server_key_iv: "bb8f4356391a205adcb452a79a722aa3"
server_key_version: "encryptionkey@0"
I20260620 22:52:00.197557 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "6339a4c09a734011876f3e0ff0b112cf"
format_stamp: "Formatted at 2026-06-20 22:52:00 on dist-test-slave-4mtw"
server_key: "7b41a339de59dea60d449a7761cde2a6"
server_key_iv: "bb8f4356391a205adcb452a79a722aa3"
server_key_version: "encryptionkey@0"
I20260620 22:52:00.198707 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:00.199332 4502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.199520 4487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:00.199597 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "6339a4c09a734011876f3e0ff0b112cf"
format_stamp: "Formatted at 2026-06-20 22:52:00 on dist-test-slave-4mtw"
server_key: "7b41a339de59dea60d449a7761cde2a6"
server_key_iv: "bb8f4356391a205adcb452a79a722aa3"
server_key_version: "encryptionkey@0"
I20260620 22:52:00.199653 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:00.213824 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:00.215404 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:00.215533 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:00.215960 4487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:00.216001 4487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.216029 4487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:00.216054 4487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:00.222605 4487 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:44699
I20260620 22:52:00.222738 4617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:44699 every 8 connection(s)
I20260620 22:52:00.222961 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:00.225127 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.216371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:50755 (local address 127.2.90.190:37577)
0620 22:52:00.216433 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.216435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.216657 (+ 222us) server_negotiation.cc:407] Connection header received
0620 22:52:00.217301 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.217302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.217315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.217367 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:00.218158 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.218679 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.219608 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.219894 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.219956 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.222861 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:00.222862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:00.222863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:00.222869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:00.224072 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.224426 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.224427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.224428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.224477 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.224631 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.224632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.224632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.224765 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:00.224812 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.225008 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.225065 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:52:00.225284 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.216355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.2.90.131:50755)
0620 22:52:00.216521 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.216567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.217202 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.217445 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.217449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.217776 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:00.218062 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.218069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.218762 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.218763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.219530 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.219533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.220001 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.220003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.220156 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.220644 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:00.220654 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:00.222770 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:00.224156 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.224161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.224163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.224371 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.224516 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.224516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.224516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.224593 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.224812 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:00.224815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:00.224917 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.224993 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.225099 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260620 22:52:00.225718 4618 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37577
I20260620 22:52:00.225792 4618 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:00.225965 4618 heartbeater.cc:511] Master 127.2.90.190:37577 requested a full tablet report, sending...
I20260620 22:52:00.226310 4134 ts_manager.cc:194] Registered new tserver with Master: 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
I20260620 22:52:00.226737 4134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:50755
I20260620 22:52:00.230625 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 4487
I20260620 22:52:00.230707 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:00.230881 2410 external_mini_cluster.cc:1428] Setting key 516b8913f473f48c276eb05d4be7c88c
I20260620 22:52:00.231335 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.226593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.2.90.131:45235)
0620 22:52:00.226722 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.226732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.226789 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.226908 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.226909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.227086 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:00.227144 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.227145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.227800 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.227800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.228298 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.228316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.228589 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.228590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.228651 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.229030 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:00.229035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:00.229274 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:00.230439 (+ 1165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.230440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.230440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.230594 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.230757 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.230758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.230758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.230809 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.231036 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:00.231037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:00.231084 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.231154 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.231246 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260620 22:52:00.231355 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.226644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:45235 (local address 127.2.90.190:37577)
0620 22:52:00.226686 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.226688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.226776 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:52:00.226833 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.226834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.226843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.226903 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:00.227210 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.227730 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.228367 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.228537 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.228577 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.229347 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:00.229349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:00.229349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:00.229355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:00.230374 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.230654 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.230655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.230656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.230711 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.230856 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.230857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.230858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.230971 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:00.231033 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.231192 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.231264 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:00.231686 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:00.252477 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.246667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.2.90.190:37577)
0620 22:52:00.246718 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.246719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.246744 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:52:00.246764 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.246765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.246773 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.246809 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:00.247291 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.247749 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.248467 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.248644 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.248716 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.250570 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:00.250571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:00.250572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:00.250578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:00.251706 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.251954 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.251955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.251955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.251998 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.252112 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.252112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.252112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.252194 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:00.252266 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.252348 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.252401 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:52:00.253798 4134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:00.255213 4134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:00.261345 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.257135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.0.0.1:52780)
0620 22:52:00.257510 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.257516 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.257541 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.257936 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.257938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.257940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.257974 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.257975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.259107 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.259107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.260365 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.260367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.261056 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.261057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.261128 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.261141 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.261217 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.261269 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":16,"thread_start_us":77,"threads_started":1}
I20260620 22:52:00.261444 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.257132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.0.0.1:33462)
0620 22:52:00.257284 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.257325 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.257387 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.257692 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.257694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.257698 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.257766 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.257767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.259026 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.259027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.260336 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.260339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.261228 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.261229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.261288 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.261291 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.261340 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.261372 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":38,"thread_start_us":48,"threads_started":1}
I20260620 22:52:00.261663 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.257357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.2.90.129:45385)
0620 22:52:00.257786 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.257790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.257798 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:00.257841 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.257843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.257856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.257927 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.258024 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.259039 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.260447 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.260985 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.261048 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.261115 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.261531 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.261573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":69,"threads_started":1}
I20260620 22:52:00.261812 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.257135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.0.0.1:40602)
0620 22:52:00.257397 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.257405 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.257444 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.257692 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.257694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.257696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.257766 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.257767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.259814 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.259815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.260665 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.260667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.261591 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.261592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.261654 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.261658 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.261699 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.261738 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260620 22:52:00.261837 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.257216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.2.90.130:42849)
0620 22:52:00.257530 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.257533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.257544 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:00.257581 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.257583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.257598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.257683 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.257835 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.258957 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.260423 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.261153 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.261222 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.261273 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.261361 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.261676 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":104,"threads_started":1}
I20260620 22:52:00.262077 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.257361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.2.90.131:44699)
0620 22:52:00.257527 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.257533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.257544 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:00.257581 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.257583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.257598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.257685 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.259061 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.259728 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.260783 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.261455 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.261512 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.261561 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.261954 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.261989 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260620 22:52:00.262300 4272 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf7bea03954febaf30b53aba1fda3a (DEFAULT_TABLE table=test-table [id=e41ff6ab1ca9436c81df48d74c0e6173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:00.262651 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf7bea03954febaf30b53aba1fda3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.262605 4552 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf7bea03954febaf30b53aba1fda3a (DEFAULT_TABLE table=test-table [id=e41ff6ab1ca9436c81df48d74c0e6173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:00.262903 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf7bea03954febaf30b53aba1fda3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.262809 4412 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf7bea03954febaf30b53aba1fda3a (DEFAULT_TABLE table=test-table [id=e41ff6ab1ca9436c81df48d74c0e6173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:00.263088 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf7bea03954febaf30b53aba1fda3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.265786 4638 tablet_bootstrap.cc:492] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Bootstrap starting.
I20260620 22:52:00.266067 4639 tablet_bootstrap.cc:492] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Bootstrap starting.
I20260620 22:52:00.266505 4638 tablet_bootstrap.cc:654] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.266839 4638 log.cc:824] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:00.266923 4639 tablet_bootstrap.cc:654] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.267189 4639 log.cc:824] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:00.267486 4640 tablet_bootstrap.cc:492] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Bootstrap starting.
I20260620 22:52:00.267573 4638 tablet_bootstrap.cc:492] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: No bootstrap required, opened a new log
I20260620 22:52:00.267633 4638 ts_tablet_manager.cc:1397] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:00.267904 4639 tablet_bootstrap.cc:492] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: No bootstrap required, opened a new log
I20260620 22:52:00.267970 4639 ts_tablet_manager.cc:1397] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:00.268049 4640 tablet_bootstrap.cc:654] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.268352 4640 log.cc:824] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:00.269043 4640 tablet_bootstrap.cc:492] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: No bootstrap required, opened a new log
I20260620 22:52:00.269114 4640 ts_tablet_manager.cc:1397] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:00.269634 4638 raft_consensus.cc:348] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.269688 4639 raft_consensus.cc:348] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.269804 4638 raft_consensus.cc:374] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.269832 4639 raft_consensus.cc:374] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.269837 4638 raft_consensus.cc:729] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d988799520a46e0ae52b2c10fe67b2c, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.269872 4639 raft_consensus.cc:729] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.269970 4638 consensus_queue.cc:260] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.269984 4639 consensus_queue.cc:260] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.270249 4639 ts_tablet_manager.cc:1428] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:00.270264 4638 ts_tablet_manager.cc:1428] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:52:00.270372 4338 heartbeater.cc:503] Master 127.2.90.190:37577 was elected leader, sending a full tablet report...
I20260620 22:52:00.270459 4618 heartbeater.cc:503] Master 127.2.90.190:37577 was elected leader, sending a full tablet report...
I20260620 22:52:00.270965 4640 raft_consensus.cc:348] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.271199 4640 raft_consensus.cc:374] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.271255 4640 raft_consensus.cc:729] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.271359 4640 consensus_queue.cc:260] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.271631 4640 ts_tablet_manager.cc:1428] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:52:00.271636 4478 heartbeater.cc:503] Master 127.2.90.190:37577 was elected leader, sending a full tablet report...
W20260620 22:52:00.331215 4479 tablet.cc:2369] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:52:00.444368 4339 tablet.cc:2369] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:00.472965 4646 raft_consensus.cc:482] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:00.473076 4646 raft_consensus.cc:504] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.473369 4646 leader_election.cc:288] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
W20260620 22:52:00.473543 4619 tablet.cc:2369] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:00.476013 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.473684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:58745 (local address 127.2.90.131:44699)
0620 22:52:00.473722 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.473723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.473730 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:00.473755 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.473756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.473766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.473822 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.473988 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.474497 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.475386 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.475722 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.475772 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.475781 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.475915 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.475948 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:00.476020 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.473733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:47535 (local address 127.2.90.129:45385)
0620 22:52:00.473799 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.473801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.473842 (+ 41us) server_negotiation.cc:407] Connection header received
0620 22:52:00.473900 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.473901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.473924 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.473975 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.474096 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.474530 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.475397 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.475738 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.475799 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.475807 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.475924 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.475955 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:52:00.476034 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.473560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.2.90.130:58745)
0620 22:52:00.473599 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.473611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.473657 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.473858 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.473859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.473861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.473930 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.473932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.474567 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.474568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.475306 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.475308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.475786 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.475787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.475854 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.475857 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.475900 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.475944 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":3}
I20260620 22:52:00.476060 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.473560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.2.90.130:47535)
0620 22:52:00.473792 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.473800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.473848 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.473982 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.473983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.473984 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.474043 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.474045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.474595 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.474596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.475317 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.475319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.475807 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.475808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.475859 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.475862 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.475908 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.475946 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
I20260620 22:52:00.476507 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf7bea03954febaf30b53aba1fda3a" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c" is_pre_election: true
I20260620 22:52:00.476529 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf7bea03954febaf30b53aba1fda3a" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339a4c09a734011876f3e0ff0b112cf" is_pre_election: true
I20260620 22:52:00.476643 4572 raft_consensus.cc:2454] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 0.
I20260620 22:52:00.476643 4292 raft_consensus.cc:2454] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 0.
I20260620 22:52:00.476859 4365 leader_election.cc:302] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, e31c02f3350a406ca0bd3b1eb81646df; no voters:
I20260620 22:52:00.477025 4646 raft_consensus.cc:2781] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:00.477063 4646 raft_consensus.cc:482] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:00.477077 4646 raft_consensus.cc:3037] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.477932 4646 raft_consensus.cc:504] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.478047 4646 leader_election.cc:288] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 election: Requested vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
I20260620 22:52:00.478161 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf7bea03954febaf30b53aba1fda3a" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c"
I20260620 22:52:00.478184 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf7bea03954febaf30b53aba1fda3a" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339a4c09a734011876f3e0ff0b112cf"
I20260620 22:52:00.478231 4292 raft_consensus.cc:3037] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.478240 4572 raft_consensus.cc:3037] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.478847 4572 raft_consensus.cc:2454] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 1.
I20260620 22:52:00.478902 4292 raft_consensus.cc:2454] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 1.
I20260620 22:52:00.479031 4363 leader_election.cc:302] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6339a4c09a734011876f3e0ff0b112cf, e31c02f3350a406ca0bd3b1eb81646df; no voters:
I20260620 22:52:00.479121 4646 raft_consensus.cc:2781] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:00.479236 4646 raft_consensus.cc:686] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 1 LEADER]: Becoming Leader. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Running, Role: LEADER
I20260620 22:52:00.479323 4646 consensus_queue.cc:237] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.480053 4135 catalog_manager.cc:5495] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df reported cstate change: term changed from 0 to 1, leader changed from <none> to e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130). New cstate: current_term: 1 leader_uuid: "e31c02f3350a406ca0bd3b1eb81646df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } health_report { overall_health: UNKNOWN } } }
Jun 20 22:52:00 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:00.507957 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.501334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.2.90.190:37577)
0620 22:52:00.501395 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.501398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.501507 (+ 109us) server_negotiation.cc:407] Connection header received
0620 22:52:00.501563 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.501564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.501573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.501636 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:00.502138 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.502913 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.503578 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.503754 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.503807 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.505955 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:00.505956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:00.505957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:00.505963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:00.506970 (+ 1007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.507277 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.507278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.507278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.507329 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:00.507486 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:00.507486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:00.507487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:00.507584 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:00.507634 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.507763 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.507842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:52:00.508011 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.501286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37577 (local address 127.0.0.1:59888)
0620 22:52:00.501438 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.501448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.501517 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.501645 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.501646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.501991 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:00.502044 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.502047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.503003 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.503004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.503520 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.503522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.503814 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.503815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.503876 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.504413 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:00.504419 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:00.505870 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:00.507035 (+ 1165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.507041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.507044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.507217 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.507379 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:00.507379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:00.507380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:00.507440 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:00.507631 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:00.507638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:00.507681 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.507738 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.507893 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":64,"thread_start_us":91,"threads_started":1}
I20260620 22:52:00.509644 4135 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:59888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:52:00.509815 4135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:00.511075 4552 tablet_service.cc:1467] Processing CreateTablet for tablet 56348938186d4e6eb34285d410d68745 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=659c71eff15d4661b4356e72a13e28ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:52:00.511068 4272 tablet_service.cc:1467] Processing CreateTablet for tablet 56348938186d4e6eb34285d410d68745 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=659c71eff15d4661b4356e72a13e28ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:52:00.511071 4412 tablet_service.cc:1467] Processing CreateTablet for tablet 56348938186d4e6eb34285d410d68745 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=659c71eff15d4661b4356e72a13e28ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:52:00.511220 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56348938186d4e6eb34285d410d68745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.511220 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56348938186d4e6eb34285d410d68745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.511220 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56348938186d4e6eb34285d410d68745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.512245 4639 tablet_bootstrap.cc:492] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: Bootstrap starting.
I20260620 22:52:00.512655 4640 tablet_bootstrap.cc:492] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: Bootstrap starting.
I20260620 22:52:00.512720 4639 tablet_bootstrap.cc:654] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.513073 4640 tablet_bootstrap.cc:654] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.513353 4639 tablet_bootstrap.cc:492] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: No bootstrap required, opened a new log
I20260620 22:52:00.513391 4639 ts_tablet_manager.cc:1397] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:00.513393 4638 tablet_bootstrap.cc:492] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: Bootstrap starting.
I20260620 22:52:00.513533 4639 raft_consensus.cc:348] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.513603 4639 raft_consensus.cc:374] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.513620 4639 raft_consensus.cc:729] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.513631 4640 tablet_bootstrap.cc:492] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: No bootstrap required, opened a new log
I20260620 22:52:00.513666 4640 ts_tablet_manager.cc:1397] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:00.513664 4639 consensus_queue.cc:260] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.513725 4638 tablet_bootstrap.cc:654] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.513757 4639 ts_tablet_manager.cc:1428] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.513800 4640 raft_consensus.cc:348] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.513804 4645 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.513872 4640 raft_consensus.cc:374] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.513876 4645 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.513892 4640 raft_consensus.cc:729] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.513952 4640 consensus_queue.cc:260] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.514034 4640 ts_tablet_manager.cc:1428] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.514065 4646 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.514118 4646 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.514300 4638 tablet_bootstrap.cc:492] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: No bootstrap required, opened a new log
I20260620 22:52:00.514350 4638 ts_tablet_manager.cc:1397] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:00.514484 4638 raft_consensus.cc:348] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.514547 4638 raft_consensus.cc:374] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.514572 4638 raft_consensus.cc:729] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d988799520a46e0ae52b2c10fe67b2c, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.514617 4638 consensus_queue.cc:260] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.514714 4638 ts_tablet_manager.cc:1428] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.514762 4644 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.514850 4644 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.548913 4645 raft_consensus.cc:482] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:00.548988 4645 raft_consensus.cc:504] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.549297 4645 leader_election.cc:288] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849)
I20260620 22:52:00.551745 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.549438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.2.90.131:33603)
0620 22:52:00.549495 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.549507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.549555 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.549669 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.549670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.549671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.549730 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.549732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.550312 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.550313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.551027 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.551028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.551528 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.551529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.551597 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.551600 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.551642 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.551677 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:52:00.551745 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.549515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:33603 (local address 127.2.90.130:42849)
0620 22:52:00.549567 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.549568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.549576 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:00.549600 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.549601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.549610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.549657 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.549786 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.550242 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.551108 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.551460 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.551519 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.551528 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.551655 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.551690 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:52:00.551831 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.549514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:44813 (local address 127.2.90.129:45385)
0620 22:52:00.549555 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.549557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.549675 (+ 118us) server_negotiation.cc:407] Connection header received
0620 22:52:00.549721 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.549722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.549730 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.549774 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.549898 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.550333 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.551187 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.551538 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.551595 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.551602 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.551739 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.551772 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:00.551846 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.549438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.2.90.131:44813)
0620 22:52:00.549611 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.549619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.549670 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.549784 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.549784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.549786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.549845 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.549846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.550401 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.550402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.551108 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.551110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.551624 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.551626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.551680 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.551683 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.551725 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.551761 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":11,"thread_start_us":98,"threads_started":1}
I20260620 22:52:00.552028 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56348938186d4e6eb34285d410d68745" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31c02f3350a406ca0bd3b1eb81646df" is_pre_election: true
I20260620 22:52:00.552053 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56348938186d4e6eb34285d410d68745" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c" is_pre_election: true
I20260620 22:52:00.552124 4292 raft_consensus.cc:2454] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 0.
I20260620 22:52:00.552138 4432 raft_consensus.cc:2454] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 0.
I20260620 22:52:00.552287 4505 leader_election.cc:302] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, 6339a4c09a734011876f3e0ff0b112cf; no voters:
I20260620 22:52:00.552417 4645 raft_consensus.cc:2781] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:00.552448 4645 raft_consensus.cc:482] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:00.552467 4645 raft_consensus.cc:3037] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.553097 4645 raft_consensus.cc:504] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.553211 4645 leader_election.cc:288] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 election: Requested vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849)
I20260620 22:52:00.553355 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56348938186d4e6eb34285d410d68745" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c"
I20260620 22:52:00.553362 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56348938186d4e6eb34285d410d68745" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31c02f3350a406ca0bd3b1eb81646df"
I20260620 22:52:00.553426 4292 raft_consensus.cc:3037] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.553426 4432 raft_consensus.cc:3037] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.553941 4292 raft_consensus.cc:2454] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 1.
I20260620 22:52:00.554006 4432 raft_consensus.cc:2454] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 1.
I20260620 22:52:00.554088 4505 leader_election.cc:302] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, 6339a4c09a734011876f3e0ff0b112cf; no voters:
I20260620 22:52:00.554193 4645 raft_consensus.cc:2781] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:00.554320 4645 raft_consensus.cc:686] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [term 1 LEADER]: Becoming Leader. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Running, Role: LEADER
I20260620 22:52:00.554407 4645 consensus_queue.cc:237] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.554812 4660 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: TxnStatusTablet state changed. Reason: New leader 6339a4c09a734011876f3e0ff0b112cf. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.554894 4660 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:52:00.555061 4662 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:52:00.555063 4135 catalog_manager.cc:5495] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131). New cstate: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } health_report { overall_health: HEALTHY } } }
I20260620 22:52:00.555176 4662 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:52:00.560876 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.558314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.0.0.1:40608)
0620 22:52:00.558394 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.558404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.558452 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.558587 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.558587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.558588 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:00.558643 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.558646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.559195 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.559196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.560072 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.560075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.560393 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.560394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.560457 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.560474 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.560692 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.560698 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.560744 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.560798 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 22:52:00.561065 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.558355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.2.90.131:44699)
0620 22:52:00.558387 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.558389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.558435 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:52:00.558518 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.558519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.558531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.558580 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.558698 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.559129 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.560158 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.560333 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.560373 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.560552 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.560629 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.560674 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.560978 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.561006 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:52:00.562538 4292 raft_consensus.cc:1261] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Refusing update from remote peer 6339a4c09a734011876f3e0ff0b112cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:00.562615 4432 raft_consensus.cc:1261] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Refusing update from remote peer 6339a4c09a734011876f3e0ff0b112cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:00.562775 4660 consensus_queue.cc:1035] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:00.562879 4645 consensus_queue.cc:1035] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:00.563889 4644 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: TxnStatusTablet state changed. Reason: New leader 6339a4c09a734011876f3e0ff0b112cf. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.563968 4644 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.563941 4646 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: TxnStatusTablet state changed. Reason: New leader 6339a4c09a734011876f3e0ff0b112cf. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.564013 4646 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.564738 4645 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.564808 4645 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:52:00.564787 4646 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.564857 4646 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P e31c02f3350a406ca0bd3b1eb81646df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.565146 4645 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.565210 4645 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 6339a4c09a734011876f3e0ff0b112cf: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:52:00.565305 4667 mvcc.cc:205] Tried to move back new op lower bound from 7299055290622119936 to 7299055290591322112. Current Snapshot: MvccSnapshot[applied={T|T < 7299055290622119936}]
I20260620 22:52:00.565601 4644 tablet_replica.cc:439] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } } }
I20260620 22:52:00.565673 4644 tablet_replica.cc:442] T 56348938186d4e6eb34285d410d68745 P 1d988799520a46e0ae52b2c10fe67b2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:00.565851 4665 mvcc.cc:205] Tried to move back new op lower bound from 7299055290622119936 to 7299055290591322112. Current Snapshot: MvccSnapshot[applied={T|T < 7299055290622119936}]
I20260620 22:52:00.569500 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.567316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.2.90.130:42849)
0620 22:52:00.567350 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.567352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.567359 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:00.567414 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.567414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.567429 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.567490 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.567616 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.568058 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.568797 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.568999 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.569049 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.569191 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.569313 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.569375 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.569424 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.569441 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:00.574025 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.571448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.2.90.130:54027)
0620 22:52:00.571510 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.571520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.571555 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.571751 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.571751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.571752 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:00.571793 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.571794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.572600 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.572600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.573294 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.573296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.573726 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.573727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.573768 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.573780 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.573926 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.573926 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.573955 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.573981 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:52:00.574039 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.571606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:54027 (local address 127.2.90.131:44699)
0620 22:52:00.571656 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.571658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.571666 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:00.571685 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.571685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.571695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.571755 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.571836 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.572537 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.573367 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.573664 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.573730 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.573850 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.573880 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.573926 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.573959 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.573976 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:52:00.575887 4292 raft_consensus.cc:1261] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Refusing update from remote peer e31c02f3350a406ca0bd3b1eb81646df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:00.575919 4572 raft_consensus.cc:1261] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Refusing update from remote peer e31c02f3350a406ca0bd3b1eb81646df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:00.576097 4648 consensus_queue.cc:1035] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:00.576217 4646 consensus_queue.cc:1035] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [LEADER]: Connected to new peer: Peer: permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:00.577661 4666 mvcc.cc:205] Tried to move back new op lower bound from 7299055290677215232 to 7299055290283765760. Current Snapshot: MvccSnapshot[applied={T|T < 7299055290677215232}]
I20260620 22:52:00.583974 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.581423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.2.90.131:53985)
0620 22:52:00.581509 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.581518 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.581558 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.581700 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.581700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.581701 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:00.581759 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.581760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.582288 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.582289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.583200 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.583202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.583607 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.583608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.583673 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.583688 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.583864 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.583866 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.583899 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.583929 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 22:52:00.583984 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.581457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:53985 (local address 127.2.90.130:42849)
0620 22:52:00.581539 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.581541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.581549 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:00.581623 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.581624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.581633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.581671 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.581821 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.582232 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.583271 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.583558 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.583606 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.583783 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.583816 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.583863 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.583903 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.583921 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 22:52:00.589887 4134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:00.589967 4134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:00.590548 4134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Sending DeleteTablet for 3 replicas of tablet a3bf7bea03954febaf30b53aba1fda3a
I20260620 22:52:00.590812 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf7bea03954febaf30b53aba1fda3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:00 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33462
I20260620 22:52:00.590844 4552 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf7bea03954febaf30b53aba1fda3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:00 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40602
I20260620 22:52:00.591004 4272 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf7bea03954febaf30b53aba1fda3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:00 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:52780
I20260620 22:52:00.591010 4134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:00.591131 4134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:00.591216 4685 tablet_replica.cc:330] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: stopping tablet replica
I20260620 22:52:00.591313 4685 raft_consensus.cc:2229] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:00.591361 4685 raft_consensus.cc:2258] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:00.592125 4552 tablet_service.cc:1467] Processing CreateTablet for tablet 47d406d171e34e8e95061969fa92521e (DEFAULT_TABLE table=test-table [id=8e3fd4c3b1f0492fa8c7906404cf43c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:00.592125 4272 tablet_service.cc:1467] Processing CreateTablet for tablet 47d406d171e34e8e95061969fa92521e (DEFAULT_TABLE table=test-table [id=8e3fd4c3b1f0492fa8c7906404cf43c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:00.592257 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d406d171e34e8e95061969fa92521e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.592247 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d406d171e34e8e95061969fa92521e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.593494 4639 tablet_bootstrap.cc:492] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Bootstrap starting.
I20260620 22:52:00.593510 4412 tablet_service.cc:1467] Processing CreateTablet for tablet 47d406d171e34e8e95061969fa92521e (DEFAULT_TABLE table=test-table [id=8e3fd4c3b1f0492fa8c7906404cf43c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:00.593590 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d406d171e34e8e95061969fa92521e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:00.593969 4639 tablet_bootstrap.cc:654] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.594560 4639 tablet_bootstrap.cc:492] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: No bootstrap required, opened a new log
I20260620 22:52:00.594606 4639 ts_tablet_manager.cc:1397] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:00.594739 4639 raft_consensus.cc:348] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.594815 4639 raft_consensus.cc:374] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.594834 4639 raft_consensus.cc:729] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.594873 4639 consensus_queue.cc:260] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.594889 4638 tablet_bootstrap.cc:492] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Bootstrap starting.
I20260620 22:52:00.594957 4639 ts_tablet_manager.cc:1428] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:00.595219 4638 tablet_bootstrap.cc:654] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.595541 4640 tablet_bootstrap.cc:492] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Bootstrap starting.
I20260620 22:52:00.595813 4638 tablet_bootstrap.cc:492] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: No bootstrap required, opened a new log
I20260620 22:52:00.595858 4638 ts_tablet_manager.cc:1397] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:00.595865 4640 tablet_bootstrap.cc:654] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:00.595983 4638 raft_consensus.cc:348] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.596040 4638 raft_consensus.cc:374] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.596060 4638 raft_consensus.cc:729] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d988799520a46e0ae52b2c10fe67b2c, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.596093 4638 consensus_queue.cc:260] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.596170 4638 ts_tablet_manager.cc:1428] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.596410 4684 tablet_replica.cc:330] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: stopping tablet replica
I20260620 22:52:00.596504 4684 raft_consensus.cc:2229] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:00.596593 4684 raft_consensus.cc:2258] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:00.596757 4640 tablet_bootstrap.cc:492] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: No bootstrap required, opened a new log
I20260620 22:52:00.596789 4640 ts_tablet_manager.cc:1397] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:00.596916 4640 raft_consensus.cc:348] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.596977 4640 raft_consensus.cc:374] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:00.596995 4640 raft_consensus.cc:729] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Initialized, Role: FOLLOWER
I20260620 22:52:00.597046 4640 consensus_queue.cc:260] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.597121 4640 ts_tablet_manager.cc:1428] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:00.597472 4685 ts_tablet_manager.cc:1905] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:00.599136 4683 tablet_replica.cc:330] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: stopping tablet replica
I20260620 22:52:00.599232 4683 raft_consensus.cc:2229] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:00.599296 4683 raft_consensus.cc:2258] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:00.599579 4684 ts_tablet_manager.cc:1905] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:00.599658 4683 ts_tablet_manager.cc:1905] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:00.601104 4683 ts_tablet_manager.cc:1918] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:00.601172 4683 log.cc:1196] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/a3bf7bea03954febaf30b53aba1fda3a
I20260620 22:52:00.601411 4683 ts_tablet_manager.cc:1939] T a3bf7bea03954febaf30b53aba1fda3a P 6339a4c09a734011876f3e0ff0b112cf: Deleting consensus metadata
I20260620 22:52:00.601660 4684 ts_tablet_manager.cc:1918] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:00.601773 4684 log.cc:1196] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/a3bf7bea03954febaf30b53aba1fda3a
I20260620 22:52:00.601758 4119 catalog_manager.cc:4841] TS 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699): tablet a3bf7bea03954febaf30b53aba1fda3a (table test-table [id=e41ff6ab1ca9436c81df48d74c0e6173]) successfully deleted
I20260620 22:52:00.601899 4685 ts_tablet_manager.cc:1918] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:00.601964 4685 log.cc:1196] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/a3bf7bea03954febaf30b53aba1fda3a
I20260620 22:52:00.602150 4684 ts_tablet_manager.cc:1939] T a3bf7bea03954febaf30b53aba1fda3a P e31c02f3350a406ca0bd3b1eb81646df: Deleting consensus metadata
I20260620 22:52:00.602185 4685 ts_tablet_manager.cc:1939] T a3bf7bea03954febaf30b53aba1fda3a P 1d988799520a46e0ae52b2c10fe67b2c: Deleting consensus metadata
I20260620 22:52:00.602531 4121 catalog_manager.cc:4841] TS 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385): tablet a3bf7bea03954febaf30b53aba1fda3a (table test-table [id=e41ff6ab1ca9436c81df48d74c0e6173]) successfully deleted
I20260620 22:52:00.602593 4120 catalog_manager.cc:4841] TS e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849): tablet a3bf7bea03954febaf30b53aba1fda3a (table test-table [id=e41ff6ab1ca9436c81df48d74c0e6173]) successfully deleted
I20260620 22:52:00.649654 4644 raft_consensus.cc:482] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:00.649762 4644 raft_consensus.cc:504] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.650089 4644 leader_election.cc:288] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849), 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
I20260620 22:52:00.654099 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.650445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:34139 (local address 127.2.90.131:44699)
0620 22:52:00.650486 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.650488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.650615 (+ 127us) server_negotiation.cc:407] Connection header received
0620 22:52:00.650714 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.650715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.650724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.650752 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.650954 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.651659 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.653223 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.653737 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.653786 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.653795 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.653998 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.654029 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:00.654153 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.650413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:34939 (local address 127.2.90.130:42849)
0620 22:52:00.650454 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.650456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.650595 (+ 139us) server_negotiation.cc:407] Connection header received
0620 22:52:00.650696 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.650697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.650706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.650734 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:00.650937 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.651650 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.653251 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.653786 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.653834 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.653841 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.654056 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.654086 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:00.654412 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.650253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.2.90.129:34939)
0620 22:52:00.650530 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.650540 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.650623 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.650796 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.650797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.650798 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.650869 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.650871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.651749 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.651750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.653128 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.653131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.653897 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.653898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.653981 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.653985 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.654280 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.654322 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":42,"thread_start_us":71,"threads_started":1}
I20260620 22:52:00.654623 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d406d171e34e8e95061969fa92521e" candidate_uuid: "1d988799520a46e0ae52b2c10fe67b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31c02f3350a406ca0bd3b1eb81646df" is_pre_election: true
I20260620 22:52:00.654712 4432 raft_consensus.cc:2454] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d988799520a46e0ae52b2c10fe67b2c in term 0.
I20260620 22:52:00.654686 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.650301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.2.90.129:34139)
0620 22:52:00.650552 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.650562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.650643 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.650812 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.650812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.650814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:00.650887 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.650888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.651750 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.651751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.653116 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.653119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.653850 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.653851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.653921 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.653925 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.654566 (+ 641us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.654607 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":56,"thread_start_us":46,"threads_started":1}
I20260620 22:52:00.654870 4224 leader_election.cc:302] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, e31c02f3350a406ca0bd3b1eb81646df; no voters:
I20260620 22:52:00.655022 4644 raft_consensus.cc:2781] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:00.655061 4644 raft_consensus.cc:482] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:00.655078 4644 raft_consensus.cc:3037] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.655946 4644 raft_consensus.cc:504] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.656064 4644 leader_election.cc:288] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [CANDIDATE]: Term 1 election: Requested vote from peers e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849), 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
I20260620 22:52:00.656359 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d406d171e34e8e95061969fa92521e" candidate_uuid: "1d988799520a46e0ae52b2c10fe67b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339a4c09a734011876f3e0ff0b112cf" is_pre_election: true
I20260620 22:52:00.656435 4572 raft_consensus.cc:2454] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d988799520a46e0ae52b2c10fe67b2c in term 0.
I20260620 22:52:00.656392 4571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d406d171e34e8e95061969fa92521e" candidate_uuid: "1d988799520a46e0ae52b2c10fe67b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339a4c09a734011876f3e0ff0b112cf"
I20260620 22:52:00.656477 4571 raft_consensus.cc:3037] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.656782 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d406d171e34e8e95061969fa92521e" candidate_uuid: "1d988799520a46e0ae52b2c10fe67b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31c02f3350a406ca0bd3b1eb81646df"
I20260620 22:52:00.656848 4432 raft_consensus.cc:3037] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:00.657330 4571 raft_consensus.cc:2454] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d988799520a46e0ae52b2c10fe67b2c in term 1.
I20260620 22:52:00.657490 4223 leader_election.cc:302] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, 6339a4c09a734011876f3e0ff0b112cf; no voters:
I20260620 22:52:00.657533 4432 raft_consensus.cc:2454] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d988799520a46e0ae52b2c10fe67b2c in term 1.
I20260620 22:52:00.657564 4644 raft_consensus.cc:2781] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:00.657675 4644 raft_consensus.cc:686] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 LEADER]: Becoming Leader. State: Replica: 1d988799520a46e0ae52b2c10fe67b2c, State: Running, Role: LEADER
I20260620 22:52:00.657748 4644 consensus_queue.cc:237] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:00.658488 4134 catalog_manager.cc:5495] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129). New cstate: current_term: 1 leader_uuid: "1d988799520a46e0ae52b2c10fe67b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } health_report { overall_health: UNKNOWN } } }
I20260620 22:52:00.671207 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.668148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.2.90.129:45385)
0620 22:52:00.668185 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.668187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.668385 (+ 198us) server_negotiation.cc:407] Connection header received
0620 22:52:00.668426 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.668427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.668445 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.668517 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.668654 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.669370 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.670354 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.670634 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.670711 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.670860 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.671000 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.671069 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.671121 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.671139 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:52:00.677363 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.673973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:37221 (local address 127.2.90.131:44699)
0620 22:52:00.674010 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.674012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.674020 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:00.674074 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.674075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.674084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.674141 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.674285 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.674972 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.676567 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.676877 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.676949 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.677128 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.677159 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.677208 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.677269 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.677289 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:00.677410 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.673798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.2.90.129:37221)
0620 22:52:00.673957 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.673968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.674021 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.674152 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.674154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.674155 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:00.674221 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.674222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.675062 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.675063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.676472 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.676475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.676960 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.676961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.677028 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.677043 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.677218 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.677220 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.677265 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.677315 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260620 22:52:00.679605 4571 raft_consensus.cc:1261] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Refusing update from remote peer 1d988799520a46e0ae52b2c10fe67b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:00.679647 4432 raft_consensus.cc:1261] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Refusing update from remote peer 1d988799520a46e0ae52b2c10fe67b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:00.679836 4644 consensus_queue.cc:1035] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:00.680020 4644 consensus_queue.cc:1035] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:00.692518 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.689480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.2.90.131:38013)
0620 22:52:00.689519 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:00.689529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:00.689558 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:00.689789 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:00.689790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:00.689791 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:00.689833 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.689834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.690683 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.690684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.691518 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:00.691520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.692137 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.692138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:00.692190 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.692194 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.692366 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.692368 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:00.692400 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 22:52:00.692437 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:52:00.692973 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:00.689647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:38013 (local address 127.2.90.129:45385)
0620 22:52:00.689680 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:00.689682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:00.689687 (+ 5us) server_negotiation.cc:407] Connection header received
0620 22:52:00.689706 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:00.689707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:00.689714 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:00.689753 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:00.689888 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.690602 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.691766 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:00.692055 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:00.692107 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:00.692280 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:00.692313 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:00.692349 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:00.692886 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 22:52:00.692906 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:52:00.693187 4666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 22:52:02.213358 4134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:02.213462 4134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:02.214200 4134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Sending DeleteTablet for 3 replicas of tablet 47d406d171e34e8e95061969fa92521e
I20260620 22:52:02.214411 4272 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d406d171e34e8e95061969fa92521e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:02 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:52780
I20260620 22:52:02.214463 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d406d171e34e8e95061969fa92521e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:02 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33462
I20260620 22:52:02.214488 4552 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d406d171e34e8e95061969fa92521e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:02 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40602
I20260620 22:52:02.214663 4725 tablet_replica.cc:330] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: stopping tablet replica
I20260620 22:52:02.214716 4725 raft_consensus.cc:2229] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:02.214745 4725 raft_consensus.cc:2258] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:02.214841 4724 tablet_replica.cc:330] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: stopping tablet replica
I20260620 22:52:02.214885 4724 raft_consensus.cc:2229] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:02.214915 4724 raft_consensus.cc:2258] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:02.214965 4725 ts_tablet_manager.cc:1905] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:02.215131 4724 ts_tablet_manager.cc:1905] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:02.216277 4725 ts_tablet_manager.cc:1918] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:52:02.216315 4725 log.cc:1196] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/47d406d171e34e8e95061969fa92521e
I20260620 22:52:02.216454 4725 ts_tablet_manager.cc:1939] T 47d406d171e34e8e95061969fa92521e P e31c02f3350a406ca0bd3b1eb81646df: Deleting consensus metadata
I20260620 22:52:02.216711 4120 catalog_manager.cc:4841] TS e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849): tablet 47d406d171e34e8e95061969fa92521e (table test-table [id=8e3fd4c3b1f0492fa8c7906404cf43c3]) successfully deleted
I20260620 22:52:02.216814 4724 ts_tablet_manager.cc:1918] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:52:02.216863 4724 log.cc:1196] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/47d406d171e34e8e95061969fa92521e
I20260620 22:52:02.217047 4724 ts_tablet_manager.cc:1939] T 47d406d171e34e8e95061969fa92521e P 6339a4c09a734011876f3e0ff0b112cf: Deleting consensus metadata
I20260620 22:52:02.217391 4119 catalog_manager.cc:4841] TS 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699): tablet 47d406d171e34e8e95061969fa92521e (table test-table [id=8e3fd4c3b1f0492fa8c7906404cf43c3]) successfully deleted
I20260620 22:52:02.219683 4723 tablet_replica.cc:330] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: stopping tablet replica
I20260620 22:52:02.219738 4723 raft_consensus.cc:2229] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:02.219827 4723 raft_consensus.cc:2258] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:02.220021 4723 ts_tablet_manager.cc:1905] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:02.221395 4723 ts_tablet_manager.cc:1918] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 22:52:02.221447 4723 log.cc:1196] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/47d406d171e34e8e95061969fa92521e
I20260620 22:52:02.221627 4723 ts_tablet_manager.cc:1939] T 47d406d171e34e8e95061969fa92521e P 1d988799520a46e0ae52b2c10fe67b2c: Deleting consensus metadata
I20260620 22:52:02.221985 4121 catalog_manager.cc:4841] TS 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385): tablet 47d406d171e34e8e95061969fa92521e (table test-table [id=8e3fd4c3b1f0492fa8c7906404cf43c3]) successfully deleted
Jun 20 22:52:02 dist-test-slave-4mtw krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:52:02 dist-test-slave-4mtw krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:02.237459 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:02.230984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.2.90.190:37577)
0620 22:52:02.231121 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:02.231123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:02.231131 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:02.231171 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:02.231171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:02.231181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:02.231233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:02.231707 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.232204 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.232956 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.233159 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.233218 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:02.235203 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:02.235204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:02.235205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:02.235211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:02.236431 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:02.236748 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:02.236749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:02.236750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:02.236802 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:02.236985 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:02.236986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:02.236986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:02.237091 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:02.237177 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:02.237285 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 22:52:02.237365 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 22:52:02.238729 4134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:02.238873 4134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:02.240049 4272 tablet_service.cc:1467] Processing CreateTablet for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e (DEFAULT_TABLE table=test-table [id=b783335f9c3b449e897773b6729de53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:02.240094 4412 tablet_service.cc:1467] Processing CreateTablet for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e (DEFAULT_TABLE table=test-table [id=b783335f9c3b449e897773b6729de53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:02.240167 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:02.240245 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:02.241343 4552 tablet_service.cc:1467] Processing CreateTablet for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e (DEFAULT_TABLE table=test-table [id=b783335f9c3b449e897773b6729de53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:02.241364 4734 tablet_bootstrap.cc:492] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Bootstrap starting.
I20260620 22:52:02.241393 4735 tablet_bootstrap.cc:492] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Bootstrap starting.
I20260620 22:52:02.241461 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:02.241708 4734 tablet_bootstrap.cc:654] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:02.241708 4735 tablet_bootstrap.cc:654] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:02.242266 4734 tablet_bootstrap.cc:492] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: No bootstrap required, opened a new log
I20260620 22:52:02.242316 4734 ts_tablet_manager.cc:1397] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:02.242444 4735 tablet_bootstrap.cc:492] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: No bootstrap required, opened a new log
I20260620 22:52:02.242445 4734 raft_consensus.cc:348] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.242489 4735 ts_tablet_manager.cc:1397] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:02.242518 4734 raft_consensus.cc:374] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:02.242538 4734 raft_consensus.cc:729] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d988799520a46e0ae52b2c10fe67b2c, State: Initialized, Role: FOLLOWER
I20260620 22:52:02.242601 4734 consensus_queue.cc:260] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.242601 4735 raft_consensus.cc:348] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.242653 4735 raft_consensus.cc:374] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:02.242671 4735 raft_consensus.cc:729] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Initialized, Role: FOLLOWER
I20260620 22:52:02.242705 4734 ts_tablet_manager.cc:1428] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:02.242709 4735 consensus_queue.cc:260] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.242880 4735 ts_tablet_manager.cc:1428] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:02.242985 4738 tablet_bootstrap.cc:492] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Bootstrap starting.
I20260620 22:52:02.243350 4738 tablet_bootstrap.cc:654] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:02.243861 4738 tablet_bootstrap.cc:492] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: No bootstrap required, opened a new log
I20260620 22:52:02.243913 4738 ts_tablet_manager.cc:1397] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:02.244029 4738 raft_consensus.cc:348] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.244082 4738 raft_consensus.cc:374] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:02.244099 4738 raft_consensus.cc:729] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Initialized, Role: FOLLOWER
I20260620 22:52:02.244138 4738 consensus_queue.cc:260] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.244228 4738 ts_tablet_manager.cc:1428] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:02.434859 4709 raft_consensus.cc:482] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:02.434952 4709 raft_consensus.cc:504] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.435110 4709 leader_election.cc:288] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849)
I20260620 22:52:02.435323 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff12e9e7f3bd4d7d8b97862a5e276d1e" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c" is_pre_election: true
I20260620 22:52:02.435320 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff12e9e7f3bd4d7d8b97862a5e276d1e" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31c02f3350a406ca0bd3b1eb81646df" is_pre_election: true
I20260620 22:52:02.435408 4292 raft_consensus.cc:2454] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 0.
I20260620 22:52:02.435420 4432 raft_consensus.cc:2454] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 0.
I20260620 22:52:02.435551 4505 leader_election.cc:302] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, 6339a4c09a734011876f3e0ff0b112cf; no voters:
I20260620 22:52:02.435653 4709 raft_consensus.cc:2781] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:02.435701 4709 raft_consensus.cc:482] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:02.435719 4709 raft_consensus.cc:3037] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:02.436342 4709 raft_consensus.cc:504] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.436442 4709 leader_election.cc:288] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 election: Requested vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849)
I20260620 22:52:02.436569 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff12e9e7f3bd4d7d8b97862a5e276d1e" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c"
I20260620 22:52:02.436586 4432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff12e9e7f3bd4d7d8b97862a5e276d1e" candidate_uuid: "6339a4c09a734011876f3e0ff0b112cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31c02f3350a406ca0bd3b1eb81646df"
I20260620 22:52:02.436638 4292 raft_consensus.cc:3037] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:02.436640 4432 raft_consensus.cc:3037] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:02.437214 4292 raft_consensus.cc:2454] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 1.
I20260620 22:52:02.437217 4432 raft_consensus.cc:2454] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6339a4c09a734011876f3e0ff0b112cf in term 1.
I20260620 22:52:02.437366 4505 leader_election.cc:302] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, 6339a4c09a734011876f3e0ff0b112cf; no voters:
I20260620 22:52:02.437453 4709 raft_consensus.cc:2781] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:02.437511 4709 raft_consensus.cc:686] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 1 LEADER]: Becoming Leader. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Running, Role: LEADER
I20260620 22:52:02.437559 4709 consensus_queue.cc:237] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.438136 4134 catalog_manager.cc:5495] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131). New cstate: current_term: 1 leader_uuid: "6339a4c09a734011876f3e0ff0b112cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } health_report { overall_health: HEALTHY } } }
I20260620 22:52:02.470197 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:02.467824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.2.90.131:44699)
0620 22:52:02.467984 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:02.467986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:02.467991 (+ 5us) server_negotiation.cc:407] Connection header received
0620 22:52:02.468007 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:02.468007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:02.468017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:02.468069 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:02.468204 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.468796 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.469529 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.469779 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.469831 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:02.469975 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:02.470008 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:02.470048 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:02.470095 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 22:52:02.470119 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260620 22:52:02.474028 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:02.472144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:55401 (local address 127.2.90.131:44699)
0620 22:52:02.472186 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:02.472187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:02.472194 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:02.472250 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:02.472251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:02.472259 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:02.472299 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:02.472372 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.472841 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.473496 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.473662 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.473703 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:02.473859 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:02.473880 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:02.473908 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:02.473940 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 22:52:02.473955 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":26}
I20260620 22:52:02.474045 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:02.472013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.2.90.131:55401)
0620 22:52:02.472132 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:02.472142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:02.472189 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:02.472305 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:02.472305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:02.472307 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:02.472339 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:02.472341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.472892 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.472892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:02.473442 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:02.473443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.473739 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.473740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:02.473788 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:02.473791 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:02.473909 (+ 118us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:02.473910 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:52:02.473938 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 22:52:02.473971 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260620 22:52:02.475596 4432 raft_consensus.cc:1261] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Refusing update from remote peer 6339a4c09a734011876f3e0ff0b112cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:02.475622 4292 raft_consensus.cc:1261] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Refusing update from remote peer 6339a4c09a734011876f3e0ff0b112cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:02.475770 4709 consensus_queue.cc:1035] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:02.475862 4704 consensus_queue.cc:1035] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:02.485386 4134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:02.485466 4134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:02.485999 4134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Sending DeleteTablet for 3 replicas of tablet ff12e9e7f3bd4d7d8b97862a5e276d1e
I20260620 22:52:02.486217 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:02 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33462
I20260620 22:52:02.486234 4552 tablet_service.cc:1514] Processing DeleteTablet for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:02 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40602
I20260620 22:52:02.486312 4272 tablet_service.cc:1514] Processing DeleteTablet for tablet ff12e9e7f3bd4d7d8b97862a5e276d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:02 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:52780
I20260620 22:52:02.486317 4725 tablet_replica.cc:330] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: stopping tablet replica
I20260620 22:52:02.486384 4725 raft_consensus.cc:2229] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:02.486393 4724 tablet_replica.cc:330] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: stopping tablet replica
I20260620 22:52:02.486429 4725 raft_consensus.cc:2258] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:02.486438 4724 raft_consensus.cc:2229] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:02.486514 4724 raft_consensus.cc:2258] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:02.486591 4723 tablet_replica.cc:330] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: stopping tablet replica
I20260620 22:52:02.486558 4134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 22:52:02.486642 4723 raft_consensus.cc:2229] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:02.486670 4723 raft_consensus.cc:2258] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:52:02.486668 4134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:02.487008 4724 ts_tablet_manager.cc:1905] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:02.487025 4723 ts_tablet_manager.cc:1905] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:02.488618 4724 ts_tablet_manager.cc:1918] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:02.488631 4723 ts_tablet_manager.cc:1918] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:02.488696 4723 log.cc:1196] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/ff12e9e7f3bd4d7d8b97862a5e276d1e
I20260620 22:52:02.488669 4724 log.cc:1196] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/ff12e9e7f3bd4d7d8b97862a5e276d1e
I20260620 22:52:02.488905 4723 ts_tablet_manager.cc:1939] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 1d988799520a46e0ae52b2c10fe67b2c: Deleting consensus metadata
I20260620 22:52:02.488924 4724 ts_tablet_manager.cc:1939] T ff12e9e7f3bd4d7d8b97862a5e276d1e P 6339a4c09a734011876f3e0ff0b112cf: Deleting consensus metadata
I20260620 22:52:02.488974 4412 tablet_service.cc:1467] Processing CreateTablet for tablet b304729a1bae461f8c6cd0d07623111e (DEFAULT_TABLE table=test-table [id=0d7e2093cdf445b4a680c8787eecb52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:02.489060 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b304729a1bae461f8c6cd0d07623111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:02.489208 4119 catalog_manager.cc:4841] TS 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699): tablet ff12e9e7f3bd4d7d8b97862a5e276d1e (table test-table [id=b783335f9c3b449e897773b6729de53c]) successfully deleted
I20260620 22:52:02.489329 4552 tablet_service.cc:1467] Processing CreateTablet for tablet b304729a1bae461f8c6cd0d07623111e (DEFAULT_TABLE table=test-table [id=0d7e2093cdf445b4a680c8787eecb52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:02.489409 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b304729a1bae461f8c6cd0d07623111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:02.490294 4735 tablet_bootstrap.cc:492] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: Bootstrap starting.
I20260620 22:52:02.490399 4738 tablet_bootstrap.cc:492] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: Bootstrap starting.
I20260620 22:52:02.490561 4121 catalog_manager.cc:4841] TS 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385): tablet ff12e9e7f3bd4d7d8b97862a5e276d1e (table test-table [id=b783335f9c3b449e897773b6729de53c]) successfully deleted
I20260620 22:52:02.490590 4735 tablet_bootstrap.cc:654] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:02.490676 4738 tablet_bootstrap.cc:654] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:02.490832 4272 tablet_service.cc:1467] Processing CreateTablet for tablet b304729a1bae461f8c6cd0d07623111e (DEFAULT_TABLE table=test-table [id=0d7e2093cdf445b4a680c8787eecb52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:02.490923 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b304729a1bae461f8c6cd0d07623111e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:02.491263 4735 tablet_bootstrap.cc:492] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: No bootstrap required, opened a new log
I20260620 22:52:02.491309 4735 ts_tablet_manager.cc:1397] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:02.491403 4735 raft_consensus.cc:348] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.491469 4735 raft_consensus.cc:374] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:02.491488 4735 raft_consensus.cc:729] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Initialized, Role: FOLLOWER
I20260620 22:52:02.491528 4735 consensus_queue.cc:260] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.491606 4735 ts_tablet_manager.cc:1428] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:02.491745 4734 tablet_bootstrap.cc:492] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c: Bootstrap starting.
I20260620 22:52:02.492058 4734 tablet_bootstrap.cc:654] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:02.492247 4738 tablet_bootstrap.cc:492] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: No bootstrap required, opened a new log
I20260620 22:52:02.492293 4738 ts_tablet_manager.cc:1397] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:02.492403 4738 raft_consensus.cc:348] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.492455 4738 raft_consensus.cc:374] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:02.492471 4738 raft_consensus.cc:729] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6339a4c09a734011876f3e0ff0b112cf, State: Initialized, Role: FOLLOWER
I20260620 22:52:02.492508 4738 consensus_queue.cc:260] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.492584 4738 ts_tablet_manager.cc:1428] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:02.492590 4734 tablet_bootstrap.cc:492] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c: No bootstrap required, opened a new log
I20260620 22:52:02.492625 4734 ts_tablet_manager.cc:1397] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:02.492733 4734 raft_consensus.cc:348] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.492792 4734 raft_consensus.cc:374] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:02.492820 4734 raft_consensus.cc:729] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d988799520a46e0ae52b2c10fe67b2c, State: Initialized, Role: FOLLOWER
I20260620 22:52:02.492856 4734 consensus_queue.cc:260] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.492933 4734 ts_tablet_manager.cc:1428] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:02.496862 4725 ts_tablet_manager.cc:1905] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:02.498247 4725 ts_tablet_manager.cc:1918] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:02.498292 4725 log.cc:1196] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/ff12e9e7f3bd4d7d8b97862a5e276d1e
I20260620 22:52:02.498462 4725 ts_tablet_manager.cc:1939] T ff12e9e7f3bd4d7d8b97862a5e276d1e P e31c02f3350a406ca0bd3b1eb81646df: Deleting consensus metadata
I20260620 22:52:02.498714 4120 catalog_manager.cc:4841] TS e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130:42849): tablet ff12e9e7f3bd4d7d8b97862a5e276d1e (table test-table [id=b783335f9c3b449e897773b6729de53c]) successfully deleted
I20260620 22:52:02.786471 4739 raft_consensus.cc:482] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:02.786562 4739 raft_consensus.cc:504] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.786724 4739 leader_election.cc:288] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
I20260620 22:52:02.786918 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b304729a1bae461f8c6cd0d07623111e" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c" is_pre_election: true
I20260620 22:52:02.786929 4571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b304729a1bae461f8c6cd0d07623111e" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339a4c09a734011876f3e0ff0b112cf" is_pre_election: true
I20260620 22:52:02.787034 4292 raft_consensus.cc:2454] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 0.
I20260620 22:52:02.787034 4571 raft_consensus.cc:2454] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 0.
I20260620 22:52:02.787166 4365 leader_election.cc:302] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, e31c02f3350a406ca0bd3b1eb81646df; no voters:
I20260620 22:52:02.787277 4739 raft_consensus.cc:2781] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:02.787333 4739 raft_consensus.cc:482] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:02.787346 4739 raft_consensus.cc:3037] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:02.788017 4739 raft_consensus.cc:504] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.788142 4739 leader_election.cc:288] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 election: Requested vote from peers 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385), 6339a4c09a734011876f3e0ff0b112cf (127.2.90.131:44699)
I20260620 22:52:02.788308 4292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b304729a1bae461f8c6cd0d07623111e" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d988799520a46e0ae52b2c10fe67b2c"
I20260620 22:52:02.788400 4292 raft_consensus.cc:3037] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:02.788352 4571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b304729a1bae461f8c6cd0d07623111e" candidate_uuid: "e31c02f3350a406ca0bd3b1eb81646df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6339a4c09a734011876f3e0ff0b112cf"
I20260620 22:52:02.788436 4571 raft_consensus.cc:3037] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:02.789187 4292 raft_consensus.cc:2454] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 1.
I20260620 22:52:02.789187 4571 raft_consensus.cc:2454] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31c02f3350a406ca0bd3b1eb81646df in term 1.
I20260620 22:52:02.789346 4365 leader_election.cc:302] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d988799520a46e0ae52b2c10fe67b2c, 6339a4c09a734011876f3e0ff0b112cf, e31c02f3350a406ca0bd3b1eb81646df; no voters:
I20260620 22:52:02.789456 4739 raft_consensus.cc:2781] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:02.789592 4739 raft_consensus.cc:686] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 1 LEADER]: Becoming Leader. State: Replica: e31c02f3350a406ca0bd3b1eb81646df, State: Running, Role: LEADER
I20260620 22:52:02.789640 4739 consensus_queue.cc:237] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } }
I20260620 22:52:02.790177 4134 catalog_manager.cc:5495] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df reported cstate change: term changed from 0 to 1, leader changed from <none> to e31c02f3350a406ca0bd3b1eb81646df (127.2.90.130). New cstate: current_term: 1 leader_uuid: "e31c02f3350a406ca0bd3b1eb81646df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31c02f3350a406ca0bd3b1eb81646df" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 } health_report { overall_health: UNKNOWN } } }
I20260620 22:52:02.845562 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:02.843404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.2.90.130:42849)
0620 22:52:02.843555 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:02.843556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:02.843563 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:02.843580 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:02.843580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:02.843590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:02.843648 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:02.843753 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.844232 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.844935 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:02.845138 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:02.845191 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:02.845330 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:02.845362 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:02.845404 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:02.845456 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:52:02.845475 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260620 22:52:02.847456 4292 raft_consensus.cc:1261] T b304729a1bae461f8c6cd0d07623111e P 1d988799520a46e0ae52b2c10fe67b2c [term 1 FOLLOWER]: Refusing update from remote peer e31c02f3350a406ca0bd3b1eb81646df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:02.847486 4571 raft_consensus.cc:1261] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Refusing update from remote peer e31c02f3350a406ca0bd3b1eb81646df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:02.847735 4753 consensus_queue.cc:1035] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d988799520a46e0ae52b2c10fe67b2c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:02.847839 4739 consensus_queue.cc:1035] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [LEADER]: Connected to new peer: Peer: permanent_uuid: "6339a4c09a734011876f3e0ff0b112cf" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:52:02.851441 4706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 22:52:04.367367 4135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:04.367466 4135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:04.368198 4135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b528a4888794018ba1e574ac463822b: Sending DeleteTablet for 3 replicas of tablet b304729a1bae461f8c6cd0d07623111e
I20260620 22:52:04.368407 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet b304729a1bae461f8c6cd0d07623111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:04 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33462
I20260620 22:52:04.368414 4272 tablet_service.cc:1514] Processing DeleteTablet for tablet b304729a1bae461f8c6cd0d07623111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:04 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:52780
I20260620 22:52:04.368453 4552 tablet_service.cc:1514] Processing DeleteTablet for tablet b304729a1bae461f8c6cd0d07623111e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:04 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40602
I20260620 22:52:04.368543 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 4207
I20260620 22:52:04.369318 4782 tablet_replica.cc:330] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: stopping tablet replica
I20260620 22:52:04.369396 4782 raft_consensus.cc:2229] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:04.369450 4780 tablet_replica.cc:330] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: stopping tablet replica
I20260620 22:52:04.369501 4780 raft_consensus.cc:2229] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:04.369606 4780 raft_consensus.cc:2258] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:04.369791 4782 raft_consensus.cc:2258] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:04.369837 4780 ts_tablet_manager.cc:1905] T b304729a1bae461f8c6cd0d07623111e P e31c02f3350a406ca0bd3b1eb81646df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:04.372983 4782 ts_tablet_manager.cc:1905] T b304729a1bae461f8c6cd0d07623111e P 6339a4c09a734011876f3e0ff0b112cf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:52:04.376428 4121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:45385 (error 108)
I20260620 22:52:04.376879 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 4347
I20260620 22:52:04.377341 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.376796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.0.0.1:52796)
0620 22:52:04.376896 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.376940 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:45385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":60,"threads_started":1}
W20260620 22:52:04.377496 4121 catalog_manager.cc:4584] TS 1d988799520a46e0ae52b2c10fe67b2c (127.2.90.129:45385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b304729a1bae461f8c6cd0d07623111e: Network error: Client connection negotiation failed: client connection to 127.2.90.129:45385: connect: Connection refused (error 111)
I20260620 22:52:04.423285 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 4487
I20260620 22:52:04.423326 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.423211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.0.0.1:33478)
0620 22:52:04.423263 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.423290 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:42849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:52:04.429975 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.429857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.0.0.1:52810)
0620 22:52:04.429918 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.429944 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:45385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:52:04.430289 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.430223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:45385 (local address 127.0.0.1:52820)
0620 22:52:04.430247 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.430270 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:45385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
I20260620 22:52:04.467084 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.466900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.0.0.1:33494)
0620 22:52:04.466983 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.467035 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:42849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 22:52:04.467552 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.467468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42849 (local address 127.0.0.1:33496)
0620 22:52:04.467506 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.467532 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:42849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:52:04.470676 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:04.470565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44699 (local address 127.0.0.1:40638)
0620 22:52:04.470615 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:04.470643 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.131:44699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:52:04.470783 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 4103
2026-06-20T22:52:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5903 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:06 dist-test-slave-4mtw krb5kdc[4787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:06 dist-test-slave-4mtw krb5kdc[4787](info): set up 2 sockets
Jun 20 22:52:06 dist-test-slave-4mtw krb5kdc[4787](info): commencing operation
krb5kdc: starting...
W20260620 22:52:07.792559 2410 mini_kdc.cc:121] Time spent starting KDC: real 3.302s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:07 dist-test-slave-4mtw krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:07.935034 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41817
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:08.011925 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:08.012131 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:08.012161 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:08.013329 4800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:08.013368 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:08.013386 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:08.013404 4800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:08.013418 4800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:08.014827 4800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:41817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:08.015064 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:08.015259 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:08.019842 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.019876 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.019872 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:08.019925 4800 server_base.cc:1034] running on GCE node
I20260620 22:52:08.020196 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:08.020468 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:08.021600 4800 hybrid_clock.cc:648] HybridClock initialized: now 1781995928021584 us; error 34 us; skew 500 ppm
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:08.023890 4800 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:08.024287 4800 webserver.cc:460] Webserver started at http://127.2.90.190:42953/ using document root <none> and password file <none>
I20260620 22:52:08.024474 4800 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:08.024514 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:08.024632 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:08.025626 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "b019fd3e87af406bac0f2a60263addf4"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "62b3eca02dc6563f72eddd9b9f4b8222"
server_key_iv: "a36b5692801efb79512cb1da097494cb"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.025944 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "b019fd3e87af406bac0f2a60263addf4"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "62b3eca02dc6563f72eddd9b9f4b8222"
server_key_iv: "a36b5692801efb79512cb1da097494cb"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.027081 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:08.027695 4815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.027827 4800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:08.027900 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "b019fd3e87af406bac0f2a60263addf4"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "62b3eca02dc6563f72eddd9b9f4b8222"
server_key_iv: "a36b5692801efb79512cb1da097494cb"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.027961 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:08.042788 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:08.044759 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:08.044883 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:08.048380 4800 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:41817
I20260620 22:52:08.048422 4867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:41817 every 8 connection(s)
I20260620 22:52:08.048734 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:08.049500 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:08.051697 4868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4: Bootstrap starting.
I20260620 22:52:08.052239 4868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:08.052497 4868 log.cc:824] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:08.053102 4868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4: No bootstrap required, opened a new log
I20260620 22:52:08.054646 4868 raft_consensus.cc:348] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019fd3e87af406bac0f2a60263addf4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41817 } }
I20260620 22:52:08.054786 4868 raft_consensus.cc:374] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:08.054818 4868 raft_consensus.cc:729] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b019fd3e87af406bac0f2a60263addf4, State: Initialized, Role: FOLLOWER
I20260620 22:52:08.054956 4868 consensus_queue.cc:260] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019fd3e87af406bac0f2a60263addf4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41817 } }
I20260620 22:52:08.055030 4868 raft_consensus.cc:388] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:08.055064 4868 raft_consensus.cc:482] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:08.055110 4868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:08.055578 4868 raft_consensus.cc:504] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019fd3e87af406bac0f2a60263addf4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41817 } }
I20260620 22:52:08.055665 4868 leader_election.cc:302] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b019fd3e87af406bac0f2a60263addf4; no voters:
I20260620 22:52:08.055819 4868 leader_election.cc:288] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:08.055857 4871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:08.056051 4868 sys_catalog.cc:564] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:08.056155 4871 raft_consensus.cc:686] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [term 1 LEADER]: Becoming Leader. State: Replica: b019fd3e87af406bac0f2a60263addf4, State: Running, Role: LEADER
I20260620 22:52:08.056269 4871 consensus_queue.cc:237] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019fd3e87af406bac0f2a60263addf4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41817 } }
I20260620 22:52:08.056610 4872 sys_catalog.cc:455] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b019fd3e87af406bac0f2a60263addf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019fd3e87af406bac0f2a60263addf4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41817 } } }
I20260620 22:52:08.056624 4873 sys_catalog.cc:455] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b019fd3e87af406bac0f2a60263addf4. Latest consensus state: current_term: 1 leader_uuid: "b019fd3e87af406bac0f2a60263addf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019fd3e87af406bac0f2a60263addf4" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41817 } } }
I20260620 22:52:08.056720 4872 sys_catalog.cc:458] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:08.056725 4873 sys_catalog.cc:458] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:08.056934 4880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:08.057415 4880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:08.058713 4880 catalog_manager.cc:1347] Generated new cluster ID: 3ae5c668c8ea4aa59e0066200059101c
I20260620 22:52:08.058763 4880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:08.059003 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 4800
I20260620 22:52:08.059106 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:08.059283 2410 external_mini_cluster.cc:1428] Setting key 4899c68a07ec7c1558c7f7b1b561a808
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:08.071141 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.059709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.2.90.190:41817)
0620 22:52:08.060973 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:08.060981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:08.061010 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:52:08.061115 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:08.061121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:08.061170 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:08.061283 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:08.062022 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.063397 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.064449 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.065069 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.065157 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.067775 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:08.067779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:08.067780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:08.067803 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:08.069646 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.069985 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.069988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.069988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.070091 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.070254 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.070254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.070254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.070556 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:08.070700 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:08.070776 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:52:08.070907 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1225,"thread_start_us":61,"threads_started":1}
I20260620 22:52:08.085037 4880 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:08.085528 4880 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:08.089001 4880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b019fd3e87af406bac0f2a60263addf4: Generated new TSK 0
I20260620 22:52:08.089135 4880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:08.112885 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41817
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:08.189992 4894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:08.190176 4894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:08.190196 4894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:08.191308 4894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:08.191347 4894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:08.191373 4894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:08.192775 4894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:08.192976 4894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:08.193166 4894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:08.197549 4900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.197561 4902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.197561 4899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:08.197767 4894 server_base.cc:1034] running on GCE node
I20260620 22:52:08.197942 4894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:08.198122 4894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:08.199295 4894 hybrid_clock.cc:648] HybridClock initialized: now 1781995928199278 us; error 27 us; skew 500 ppm
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:08.201458 4894 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:08.201834 4894 webserver.cc:460] Webserver started at http://127.2.90.129:46443/ using document root <none> and password file <none>
I20260620 22:52:08.202013 4894 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:08.202056 4894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:08.202171 4894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:08.203114 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "68a4ea5ab67041c5b8e7a6475e203866"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "45370fee80503a3b6f24ad222da7e27c"
server_key_iv: "1578a4ccf4fb2cf9991090fc309aa869"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.203436 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "68a4ea5ab67041c5b8e7a6475e203866"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "45370fee80503a3b6f24ad222da7e27c"
server_key_iv: "1578a4ccf4fb2cf9991090fc309aa869"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.204618 4894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:08.205312 4909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.205471 4894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:08.205559 4894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "68a4ea5ab67041c5b8e7a6475e203866"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "45370fee80503a3b6f24ad222da7e27c"
server_key_iv: "1578a4ccf4fb2cf9991090fc309aa869"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.205618 4894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:08.215986 4894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:08.217782 4894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:08.217902 4894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:08.218089 4894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:08.218360 4894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:08.218392 4894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.218418 4894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:08.218442 4894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.223680 4894 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:34049
I20260620 22:52:08.223735 5022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:34049 every 8 connection(s)
I20260620 22:52:08.224000 4894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:08.226985 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 4894
I20260620 22:52:08.227063 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:08.227206 2410 external_mini_cluster.cc:1428] Setting key 6f1d25c4aa7a1011450e8708078dc856
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:08.233086 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.224587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:40239 (local address 127.2.90.190:41817)
0620 22:52:08.224646 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:08.224648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:08.224871 (+ 223us) server_negotiation.cc:407] Connection header received
0620 22:52:08.225318 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:08.225319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:08.225328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:08.225364 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:08.226209 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.226701 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.227830 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.228051 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.228105 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.230367 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:08.230368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:08.230369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:08.230376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:08.231818 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.232169 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.232169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.232170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.232243 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.232442 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.232443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.232443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.232581 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:08.232650 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:08.232910 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 22:52:08.232995 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:52:08.233142 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.224591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41817 (local address 127.2.90.129:40239)
0620 22:52:08.224759 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:08.224796 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:08.225232 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:08.225422 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:08.225425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:08.225939 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:08.226112 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:08.226117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.226768 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.226768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:08.227756 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:08.227759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.228123 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.228125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:08.228261 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.228765 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:08.228774 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:08.230250 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:08.231892 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:08.231897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:08.231898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:08.232102 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:08.232302 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:08.232303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:08.232303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:08.232389 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:08.232636 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:08.232639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:08.232789 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 22:52:08.232871 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:52:08.232986 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260620 22:52:08.233829 5023 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41817
I20260620 22:52:08.233928 5023 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:08.234112 5023 heartbeater.cc:511] Master 127.2.90.190:41817 requested a full tablet report, sending...
I20260620 22:52:08.234571 4832 ts_manager.cc:194] Registered new tserver with Master: 68a4ea5ab67041c5b8e7a6475e203866 (127.2.90.129:34049)
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:52:08.235188 4832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:40239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:08.249157 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41817
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:08.326781 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:08.326974 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:08.326996 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:08.328110 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:08.328153 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:08.328178 5028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:08.329622 5028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:08.329841 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:08.330081 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:52:08.334800 5028 server_base.cc:1034] running on GCE node
W20260620 22:52:08.334751 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.334766 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.334766 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:08.335192 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:08.335438 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:08.336583 5028 hybrid_clock.cc:648] HybridClock initialized: now 1781995928336570 us; error 27 us; skew 500 ppm
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:08.338865 5028 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:08.339244 5028 webserver.cc:460] Webserver started at http://127.2.90.130:32911/ using document root <none> and password file <none>
I20260620 22:52:08.339450 5028 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:08.339497 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:08.339617 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:08.340601 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "26d859085f974d539974e66f87156e09"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "1fb510c25825a9b61e8f291a1129bd1a"
server_key_iv: "11e1251f6c48d76baad06dfbfab6f8ed"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.340966 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "26d859085f974d539974e66f87156e09"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "1fb510c25825a9b61e8f291a1129bd1a"
server_key_iv: "11e1251f6c48d76baad06dfbfab6f8ed"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.342126 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:08.342762 5043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.342887 5028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:08.342963 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "26d859085f974d539974e66f87156e09"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "1fb510c25825a9b61e8f291a1129bd1a"
server_key_iv: "11e1251f6c48d76baad06dfbfab6f8ed"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.343027 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:08.360891 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:08.362514 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:08.362641 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:08.362846 5028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:08.363132 5028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:08.363163 5028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.363183 5028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:08.363199 5028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.368347 5028 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:43451
I20260620 22:52:08.368407 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:43451 every 8 connection(s)
I20260620 22:52:08.368671 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:08.373100 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5028
I20260620 22:52:08.373193 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:08.373382 2410 external_mini_cluster.cc:1428] Setting key 359f3ae8720f839c34a503303b039730
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:08.377463 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.369315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:47885 (local address 127.2.90.190:41817)
0620 22:52:08.369365 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:08.369367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:08.369543 (+ 176us) server_negotiation.cc:407] Connection header received
0620 22:52:08.369986 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:08.369987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:08.369996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:08.370045 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:08.370904 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.371411 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.372333 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.372521 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.372575 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.374827 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:08.374828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:08.374828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:08.374834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:08.376051 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.376469 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.376470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.376470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.376533 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.376831 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.376832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.376832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.376966 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:08.377036 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:08.377290 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 22:52:08.377374 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:52:08.377552 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.369287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41817 (local address 127.2.90.130:47885)
0620 22:52:08.369429 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:08.369462 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:08.369910 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:08.370097 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:08.370101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:08.370624 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:08.370811 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:08.370816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.371477 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.371478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:08.372255 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:08.372257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.372578 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.372579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:08.372668 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.373135 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:08.373145 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:08.374740 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:08.376124 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:08.376128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:08.376130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:08.376395 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:08.376614 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:08.376615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:08.376615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:08.376770 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:08.377025 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:08.377027 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:08.377157 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 22:52:08.377246 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 22:52:08.377386 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260620 22:52:08.377948 5157 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41817
I20260620 22:52:08.378031 5157 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:08.378185 5157 heartbeater.cc:511] Master 127.2.90.190:41817 requested a full tablet report, sending...
I20260620 22:52:08.378535 4832 ts_manager.cc:194] Registered new tserver with Master: 26d859085f974d539974e66f87156e09 (127.2.90.130:43451)
I20260620 22:52:08.379015 4832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:47885
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:08.395074 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41817
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:08.471232 5162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:08.471431 5162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:08.471462 5162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:08.472627 5162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:08.472668 5162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:08.472723 5162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:08.474177 5162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:08.474404 5162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:08.474604 5162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:08.479043 5167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:08.479096 5168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:08.479162 5162 server_base.cc:1034] running on GCE node
W20260620 22:52:08.479054 5170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:08.479418 5162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:08.479610 5162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:08.480722 5162 hybrid_clock.cc:648] HybridClock initialized: now 1781995928480712 us; error 26 us; skew 500 ppm
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:08.482930 5162 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:08.483294 5162 webserver.cc:460] Webserver started at http://127.2.90.131:40089/ using document root <none> and password file <none>
I20260620 22:52:08.483490 5162 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:08.483531 5162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:08.483640 5162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:08.484601 5162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "d421cbf28e0b4ce1bb4b9591fdb4a055"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "b2ed7adcc2e88e78120ad61bb55a7a6e"
server_key_iv: "759d5b2d6104c782bd4b6e84d9c42507"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.484949 5162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "d421cbf28e0b4ce1bb4b9591fdb4a055"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "b2ed7adcc2e88e78120ad61bb55a7a6e"
server_key_iv: "759d5b2d6104c782bd4b6e84d9c42507"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.486071 5162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:08.486697 5177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.486842 5162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:08.486913 5162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d421cbf28e0b4ce1bb4b9591fdb4a055"
format_stamp: "Formatted at 2026-06-20 22:52:08 on dist-test-slave-4mtw"
server_key: "b2ed7adcc2e88e78120ad61bb55a7a6e"
server_key_iv: "759d5b2d6104c782bd4b6e84d9c42507"
server_key_version: "encryptionkey@0"
I20260620 22:52:08.486974 5162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:08.508498 5162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:08.510560 5162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:08.510666 5162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:08.510860 5162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:08.511147 5162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:08.511179 5162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.511201 5162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:08.511224 5162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:08.516392 5162 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:41725
I20260620 22:52:08.516463 5290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:41725 every 8 connection(s)
I20260620 22:52:08.516744 5162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:08.518824 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5162
I20260620 22:52:08.518903 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:08.519083 2410 external_mini_cluster.cc:1428] Setting key 98c750f6e8c2a4523820fc319f705044
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[4787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995928, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:08.524912 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.517394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:55411 (local address 127.2.90.190:41817)
0620 22:52:08.517451 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:08.517453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:08.517642 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:52:08.518079 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:08.518080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:08.518089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:08.518145 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:08.519033 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.519691 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.520663 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.520905 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.520958 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.522828 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:08.522829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:08.522830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:08.522836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:08.523874 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.524176 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.524176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.524177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.524220 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:08.524413 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:08.524413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:08.524414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:08.524514 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:08.524563 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:08.524793 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 22:52:08.524851 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:52:08.525080 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:08.517373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41817 (local address 127.2.90.131:55411)
0620 22:52:08.517524 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:08.517565 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:08.518009 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:08.518198 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:08.518202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:08.518746 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:08.518942 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:08.518948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.519770 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.519771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:08.520597 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:08.520600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:08.520964 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:08.520965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:08.521060 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:08.521503 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:08.521512 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:08.522752 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:08.523934 (+ 1182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:08.523938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:08.523940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:08.524125 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:08.524261 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:08.524262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:08.524262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:08.524361 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:08.524559 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:08.524562 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:08.524662 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 22:52:08.524776 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 22:52:08.524886 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260620 22:52:08.525470 5291 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41817
I20260620 22:52:08.525552 5291 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:08.525705 5291 heartbeater.cc:511] Master 127.2.90.190:41817 requested a full tablet report, sending...
I20260620 22:52:08.526023 4832 ts_manager.cc:194] Registered new tserver with Master: d421cbf28e0b4ce1bb4b9591fdb4a055 (127.2.90.131:41725)
I20260620 22:52:08.526361 4832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:55411
I20260620 22:52:08.530658 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:52:08.540071 4891 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:08.539342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.2.90.190:41817)
0620 22:52:08.539391 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:08.539393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:08.539473 (+ 80us) server_negotiation.cc:407] Connection header received
0620 22:52:08.539546 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:08.539546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:08.539554 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:08.539602 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:08.540024 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53078: BlockingRecv error: recv got EOF from 127.0.0.1:53078 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:52:08.541154 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 4894
I20260620 22:52:08.544859 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5028
I20260620 22:52:08.548326 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5162
I20260620 22:52:08.552044 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 4800
2026-06-20T22:52:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4080 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[5303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[5303](info): set up 2 sockets
Jun 20 22:52:08 dist-test-slave-4mtw krb5kdc[5303](info): commencing operation
krb5kdc: starting...
W20260620 22:52:09.496240 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:09 dist-test-slave-4mtw krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:09.601578 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39911
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:39911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:09.678891 5316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:09.679075 5316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:09.679095 5316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:09.680173 5316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:09.680203 5316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:09.680215 5316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:09.680226 5316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:09.680239 5316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:09.681650 5316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:39911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:09.681830 5316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:09.682006 5316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:52:09.686311 5316 server_base.cc:1034] running on GCE node
W20260620 22:52:09.686295 5321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:09.686278 5324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:09.686282 5322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:09.686775 5316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:09.687037 5316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:09.688167 5316 hybrid_clock.cc:648] HybridClock initialized: now 1781995929688155 us; error 29 us; skew 500 ppm
Jun 20 22:52:09 dist-test-slave-4mtw krb5kdc[5303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995929, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:09.690461 5316 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:09.690829 5316 webserver.cc:460] Webserver started at http://127.2.90.190:37125/ using document root <none> and password file <none>
I20260620 22:52:09.691008 5316 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:09.691046 5316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:09.691133 5316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:09.692081 5316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "84aaf6fe42e74f9abc00c6f77e10751c"
format_stamp: "Formatted at 2026-06-20 22:52:09 on dist-test-slave-4mtw"
server_key: "ecfca3344606af37253d7ca7c5cb8b6e"
server_key_iv: "90de04abd7324d18831f20e18ae4ba72"
server_key_version: "encryptionkey@0"
I20260620 22:52:09.692366 5316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "84aaf6fe42e74f9abc00c6f77e10751c"
format_stamp: "Formatted at 2026-06-20 22:52:09 on dist-test-slave-4mtw"
server_key: "ecfca3344606af37253d7ca7c5cb8b6e"
server_key_iv: "90de04abd7324d18831f20e18ae4ba72"
server_key_version: "encryptionkey@0"
I20260620 22:52:09.693531 5316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:09.694126 5331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:09.694269 5316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:09.694340 5316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "84aaf6fe42e74f9abc00c6f77e10751c"
format_stamp: "Formatted at 2026-06-20 22:52:09 on dist-test-slave-4mtw"
server_key: "ecfca3344606af37253d7ca7c5cb8b6e"
server_key_iv: "90de04abd7324d18831f20e18ae4ba72"
server_key_version: "encryptionkey@0"
I20260620 22:52:09.694403 5316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:09.719964 5316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:09.721844 5316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:09.721969 5316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:09.725471 5316 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:39911
I20260620 22:52:09.725509 5383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:39911 every 8 connection(s)
I20260620 22:52:09.725840 5316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:09.726620 5384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:09.728706 5384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c: Bootstrap starting.
I20260620 22:52:09.729205 5384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:09.729420 5384 log.cc:824] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:09.730007 5384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c: No bootstrap required, opened a new log
I20260620 22:52:09.731446 5384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39911 } }
I20260620 22:52:09.731572 5384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:09.731592 5384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aaf6fe42e74f9abc00c6f77e10751c, State: Initialized, Role: FOLLOWER
I20260620 22:52:09.731698 5384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39911 } }
I20260620 22:52:09.731767 5384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:09.731793 5384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:09.731830 5384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:09.732514 5384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39911 } }
I20260620 22:52:09.732594 5384 leader_election.cc:302] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84aaf6fe42e74f9abc00c6f77e10751c; no voters:
I20260620 22:52:09.732746 5384 leader_election.cc:288] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:09.732787 5387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:09.732930 5387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [term 1 LEADER]: Becoming Leader. State: Replica: 84aaf6fe42e74f9abc00c6f77e10751c, State: Running, Role: LEADER
I20260620 22:52:09.733021 5384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:09.733057 5387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39911 } }
I20260620 22:52:09.733398 5389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84aaf6fe42e74f9abc00c6f77e10751c. Latest consensus state: current_term: 1 leader_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39911 } } }
I20260620 22:52:09.733387 5388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aaf6fe42e74f9abc00c6f77e10751c" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39911 } } }
I20260620 22:52:09.733490 5389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:09.733496 5388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:09.733721 5396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:09.734272 5396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:09.735581 5396 catalog_manager.cc:1347] Generated new cluster ID: a63c3ef895474e889294fa8f1abff3c3
I20260620 22:52:09.735632 5396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:09.735762 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5316
I20260620 22:52:09.735857 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:09.736029 2410 external_mini_cluster.cc:1428] Setting key c6d6891e6c2c851d0f17568defe1a144
Jun 20 22:52:09 dist-test-slave-4mtw krb5kdc[5303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:09.746829 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:09.736574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.2.90.190:39911)
0620 22:52:09.736736 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:09.736745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:09.736775 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:52:09.736886 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:09.736892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:09.736954 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:09.737078 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:09.737763 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:09.739120 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:09.740161 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:09.740788 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:09.740876 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:09.743461 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:09.743466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:09.743466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:09.743478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:09.745317 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:09.745662 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:09.745665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:09.745665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:09.745768 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:09.745946 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:09.745947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:09.745947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:09.746240 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:09.746366 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:09.746457 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:52:09.746590 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260620 22:52:09.747766 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:52:09.749696 5396 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:09.750332 5396 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:09.757903 5396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84aaf6fe42e74f9abc00c6f77e10751c: Generated new TSK 0
I20260620 22:52:09.758061 5396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:09.764091 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:09.761026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.2.90.190:39911)
0620 22:52:09.761063 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:09.761065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:09.761182 (+ 117us) server_negotiation.cc:407] Connection header received
0620 22:52:09.761233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:09.761234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:09.761254 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:09.761298 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:09.761654 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:09.762144 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:09.762840 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:09.763056 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:09.763108 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:09.763696 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:09.763697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:09.763697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:09.763862 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:09.763877 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:09.763883 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:09.763935 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:09.763999 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:52:09.764021 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:09.766108 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5316
2026-06-20T22:52:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1212 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:09 dist-test-slave-4mtw krb5kdc[5417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:09 dist-test-slave-4mtw krb5kdc[5417](info): set up 2 sockets
Jun 20 22:52:09 dist-test-slave-4mtw krb5kdc[5417](info): commencing operation
krb5kdc: starting...
W20260620 22:52:10.708204 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:10 dist-test-slave-4mtw krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:10.814934 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42449
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:10.890754 5430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:10.890959 5430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:10.890980 5430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:10.892087 5430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:10.892128 5430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:10.892140 5430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:10.892163 5430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:10.892185 5430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:10.893646 5430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:10.893872 5430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:10.894076 5430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:10.898492 5436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:10.898447 5438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:10.898459 5435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:10.898597 5430 server_base.cc:1034] running on GCE node
I20260620 22:52:10.898890 5430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:10.899129 5430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:10.900285 5430 hybrid_clock.cc:648] HybridClock initialized: now 1781995930900271 us; error 30 us; skew 500 ppm
Jun 20 22:52:10 dist-test-slave-4mtw krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995930, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:10.906445 5430 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:10.906840 5430 webserver.cc:460] Webserver started at http://127.2.90.190:35779/ using document root <none> and password file <none>
I20260620 22:52:10.907042 5430 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:10.907083 5430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:10.907199 5430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:10.908190 5430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "c2c4fa5f4c7a4de08828b157523fc38d"
format_stamp: "Formatted at 2026-06-20 22:52:10 on dist-test-slave-4mtw"
server_key: "3a3d4cd6698c7074db64e6a1df557f9c"
server_key_iv: "cd08be524f4fdf755116a74b148d3d10"
server_key_version: "encryptionkey@0"
I20260620 22:52:10.908501 5430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "c2c4fa5f4c7a4de08828b157523fc38d"
format_stamp: "Formatted at 2026-06-20 22:52:10 on dist-test-slave-4mtw"
server_key: "3a3d4cd6698c7074db64e6a1df557f9c"
server_key_iv: "cd08be524f4fdf755116a74b148d3d10"
server_key_version: "encryptionkey@0"
I20260620 22:52:10.909704 5430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:10.910392 5445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:10.910555 5430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:10.910642 5430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "c2c4fa5f4c7a4de08828b157523fc38d"
format_stamp: "Formatted at 2026-06-20 22:52:10 on dist-test-slave-4mtw"
server_key: "3a3d4cd6698c7074db64e6a1df557f9c"
server_key_iv: "cd08be524f4fdf755116a74b148d3d10"
server_key_version: "encryptionkey@0"
I20260620 22:52:10.910709 5430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:10.929893 5430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:10.931556 5430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:10.931675 5430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:10.935278 5430 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:42449
I20260620 22:52:10.935330 5497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:42449 every 8 connection(s)
I20260620 22:52:10.935639 5430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:10.936533 5498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:10.938781 5498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d: Bootstrap starting.
I20260620 22:52:10.938995 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5430
I20260620 22:52:10.939090 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:10.939251 2410 external_mini_cluster.cc:1428] Setting key 101766fc43a65a5ef14ecc8bf57f55b6
I20260620 22:52:10.939462 5498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:10.939749 5498 log.cc:824] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:10.940407 5498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d: No bootstrap required, opened a new log
I20260620 22:52:10.942456 5498 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42449 } }
I20260620 22:52:10.942626 5498 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:10.942660 5498 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c4fa5f4c7a4de08828b157523fc38d, State: Initialized, Role: FOLLOWER
I20260620 22:52:10.942780 5498 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42449 } }
I20260620 22:52:10.942858 5498 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:10.942890 5498 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:10.942941 5498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:10.943496 5498 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42449 } }
I20260620 22:52:10.943584 5498 leader_election.cc:302] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2c4fa5f4c7a4de08828b157523fc38d; no voters:
I20260620 22:52:10.943730 5498 leader_election.cc:288] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:10.943766 5503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:10.943931 5503 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [term 1 LEADER]: Becoming Leader. State: Replica: c2c4fa5f4c7a4de08828b157523fc38d, State: Running, Role: LEADER
I20260620 22:52:10.944022 5498 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:10.944059 5503 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42449 } }
I20260620 22:52:10.944756 5505 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2c4fa5f4c7a4de08828b157523fc38d. Latest consensus state: current_term: 1 leader_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42449 } } }
I20260620 22:52:10.944890 5505 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:10.945102 5513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:10.945569 5513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:10.945576 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c4fa5f4c7a4de08828b157523fc38d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42449 } } }
I20260620 22:52:10.945680 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d [sys.catalog]: This master's current role is: LEADER
Jun 20 22:52:10 dist-test-slave-4mtw krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:10.947266 5513 catalog_manager.cc:1347] Generated new cluster ID: aa8368f7d82f40149022e56483a25558
I20260620 22:52:10.947549 5513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:10.949600 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:10.939734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.2.90.190:42449)
0620 22:52:10.939896 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:10.939904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:10.939934 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:52:10.940413 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:10.940427 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:10.940489 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:10.940613 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:10.941362 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:10.942194 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:10.943184 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:10.943605 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:10.943702 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:10.946491 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:10.946495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:10.946495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:10.946507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:10.948156 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:10.948462 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:10.948465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:10.948466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:10.948546 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:10.948755 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:10.948756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:10.948756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:10.948974 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:10.949112 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:10.949292 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 22:52:10.949418 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260620 22:52:10.971019 5513 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:10.971764 5513 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:10.977977 5513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2c4fa5f4c7a4de08828b157523fc38d: Generated new TSK 0
I20260620 22:52:10.978155 5513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:11.012037 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42449
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:11.089951 5524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:11.090137 5524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:11.090158 5524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:11.091213 5524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:11.091243 5524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:11.091272 5524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:11.092615 5524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:11.092814 5524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:11.092986 5524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:11.097368 5532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:11.097375 5530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:11.097492 5524 server_base.cc:1034] running on GCE node
W20260620 22:52:11.097445 5529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:11.097903 5524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:11.098119 5524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:11.099252 5524 hybrid_clock.cc:648] HybridClock initialized: now 1781995931099238 us; error 31 us; skew 500 ppm
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:11.101557 5524 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:11.101935 5524 webserver.cc:460] Webserver started at http://127.2.90.129:40095/ using document root <none> and password file <none>
I20260620 22:52:11.102136 5524 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:11.102178 5524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:11.102286 5524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:11.103230 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "bd039f65c4e144cb8afb4f3bbc88f126"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "de384b0576241c933d9b8baaba3934d3"
server_key_iv: "f621870825bc63b9cb43d3dafe4dd1bb"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.103540 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "bd039f65c4e144cb8afb4f3bbc88f126"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "de384b0576241c933d9b8baaba3934d3"
server_key_iv: "f621870825bc63b9cb43d3dafe4dd1bb"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.104825 5524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:11.105471 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.105618 5524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:11.105681 5524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "bd039f65c4e144cb8afb4f3bbc88f126"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "de384b0576241c933d9b8baaba3934d3"
server_key_iv: "f621870825bc63b9cb43d3dafe4dd1bb"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.105746 5524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:11.121516 5524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:11.123507 5524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:11.123623 5524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:11.123832 5524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:11.124106 5524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:11.124138 5524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.124159 5524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:11.124182 5524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.129464 5524 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:40869
I20260620 22:52:11.129525 5652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:40869 every 8 connection(s)
I20260620 22:52:11.129787 5524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:11.136746 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5524
I20260620 22:52:11.136827 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:11.136987 2410 external_mini_cluster.cc:1428] Setting key f412612f5c0e36b917b1a18090131ef9
I20260620 22:52:11.137962 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.130438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:42885 (local address 127.2.90.190:42449)
0620 22:52:11.130480 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:11.130481 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:11.130685 (+ 204us) server_negotiation.cc:407] Connection header received
0620 22:52:11.131129 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:11.131130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:11.131139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:11.131200 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:11.132012 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.132508 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.133480 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.133660 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.133701 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.135527 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:11.135529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:11.135529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:11.135535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:11.136622 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.137021 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.137022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.137023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.137084 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.137332 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.137332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.137333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.137431 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:11.137480 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:11.137780 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 22:52:11.137882 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:11.138005 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.130403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42449 (local address 127.2.90.129:42885)
0620 22:52:11.130564 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:11.130605 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:11.131054 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:11.131250 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:11.131253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:11.131765 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:11.131943 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:11.131948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.132573 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.132574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:11.133388 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:11.133390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.133713 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.133714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:11.133808 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.134208 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:11.134216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:11.135449 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:11.136719 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:11.136725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:11.136726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:11.136957 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:11.137140 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:11.137141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:11.137141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:11.137273 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:11.137486 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:11.137491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:11.137636 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 22:52:11.137739 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:52:11.137851 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260620 22:52:11.139186 5653 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42449
I20260620 22:52:11.139277 5653 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:11.139446 5653 heartbeater.cc:511] Master 127.2.90.190:42449 requested a full tablet report, sending...
I20260620 22:52:11.139925 5462 ts_manager.cc:194] Registered new tserver with Master: bd039f65c4e144cb8afb4f3bbc88f126 (127.2.90.129:40869)
I20260620 22:52:11.140517 5462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:42885
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:11.158812 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42449
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:11.236724 5658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:11.236922 5658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:11.236951 5658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:11.238040 5658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:11.238080 5658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:11.238112 5658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:11.239480 5658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:11.239694 5658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:11.239889 5658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:11.244400 5666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:11.244376 5663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:11.244392 5664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:11.244535 5658 server_base.cc:1034] running on GCE node
I20260620 22:52:11.244920 5658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:11.245154 5658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:11.246287 5658 hybrid_clock.cc:648] HybridClock initialized: now 1781995931246271 us; error 31 us; skew 500 ppm
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:11.248522 5658 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:11.248944 5658 webserver.cc:460] Webserver started at http://127.2.90.130:43927/ using document root <none> and password file <none>
I20260620 22:52:11.249130 5658 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:11.249174 5658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:11.249289 5658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:11.250241 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "86bc451dca274c21802c110ca935881b"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "10a9b151473b6cb44b984d272a9a2ab0"
server_key_iv: "ec15b5b09f9113f19ec040431f29bc12"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.250543 5658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "86bc451dca274c21802c110ca935881b"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "10a9b151473b6cb44b984d272a9a2ab0"
server_key_iv: "ec15b5b09f9113f19ec040431f29bc12"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.251679 5658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:11.252298 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.252458 5658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:11.252533 5658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "86bc451dca274c21802c110ca935881b"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "10a9b151473b6cb44b984d272a9a2ab0"
server_key_iv: "ec15b5b09f9113f19ec040431f29bc12"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.252597 5658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:11.266361 5658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:11.267869 5658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:11.267993 5658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:11.268182 5658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:11.268455 5658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:11.268486 5658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.268512 5658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:11.268528 5658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.273731 5658 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:41939
I20260620 22:52:11.273768 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:41939 every 8 connection(s)
I20260620 22:52:11.274035 5658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:11.282665 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5658
I20260620 22:52:11.282766 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:11.282851 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.274625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:46021 (local address 127.2.90.190:42449)
0620 22:52:11.274662 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:11.274664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:11.274842 (+ 178us) server_negotiation.cc:407] Connection header received
0620 22:52:11.275313 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:11.275314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:11.275325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:11.275363 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:11.276478 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.277258 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.278706 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.278915 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.278967 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.280813 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:11.280814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:11.280814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:11.280820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:11.281798 (+ 978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.282137 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.282138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.282138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.282185 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.282331 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.282331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.282332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.282417 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:11.282460 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:11.282674 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 22:52:11.282751 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:52:11.282927 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.274600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42449 (local address 127.2.90.130:46021)
0620 22:52:11.274727 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:11.274764 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:11.275220 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:11.275441 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:11.275445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:11.276125 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:11.276379 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:11.276385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.277392 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.277393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:11.278626 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:11.278628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.278982 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.278983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:11.279109 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.279476 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:11.279483 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:11.280730 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:11.281870 (+ 1140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:11.281875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:11.281876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:11.282081 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:11.282220 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:11.282221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:11.282221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:11.282299 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:11.282460 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:11.282462 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:11.282560 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 22:52:11.282633 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:52:11.282747 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260620 22:52:11.282954 2410 external_mini_cluster.cc:1428] Setting key 3a839b7b6d11469e61b2670d00b0009a
I20260620 22:52:11.283452 5787 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42449
I20260620 22:52:11.283542 5787 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:11.283706 5787 heartbeater.cc:511] Master 127.2.90.190:42449 requested a full tablet report, sending...
I20260620 22:52:11.284090 5461 ts_manager.cc:194] Registered new tserver with Master: 86bc451dca274c21802c110ca935881b (127.2.90.130:41939)
I20260620 22:52:11.284447 5461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:46021
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:11.304284 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42449
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:11.380128 5792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:11.380330 5792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:11.380362 5792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:11.381500 5792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:11.381541 5792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:11.381574 5792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:11.383015 5792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:11.383239 5792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:11.383438 5792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:11.387919 5800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:11.387925 5798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:11.387934 5797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:11.388180 5792 server_base.cc:1034] running on GCE node
I20260620 22:52:11.388350 5792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:11.388542 5792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:11.389673 5792 hybrid_clock.cc:648] HybridClock initialized: now 1781995931389644 us; error 41 us; skew 500 ppm
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:11.391920 5792 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:11.392302 5792 webserver.cc:460] Webserver started at http://127.2.90.131:43247/ using document root <none> and password file <none>
I20260620 22:52:11.392474 5792 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:11.392513 5792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:11.392601 5792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:11.393599 5792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "e0ba1ee5ae6d41b99c5fb02746dedba8"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "727af93bcc314b34db08e4a07b4e1de2"
server_key_iv: "8b157e59a77055ac356aff98a8066e93"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.393878 5792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "e0ba1ee5ae6d41b99c5fb02746dedba8"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "727af93bcc314b34db08e4a07b4e1de2"
server_key_iv: "8b157e59a77055ac356aff98a8066e93"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.395028 5792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:11.395651 5807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.395844 5792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:11.395939 5792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "e0ba1ee5ae6d41b99c5fb02746dedba8"
format_stamp: "Formatted at 2026-06-20 22:52:11 on dist-test-slave-4mtw"
server_key: "727af93bcc314b34db08e4a07b4e1de2"
server_key_iv: "8b157e59a77055ac356aff98a8066e93"
server_key_version: "encryptionkey@0"
I20260620 22:52:11.396009 5792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:11.445503 5792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:11.447254 5792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:11.447376 5792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:11.447570 5792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:11.447870 5792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:11.447901 5792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.447922 5792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:11.447935 5792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:11.453331 5792 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:33825
I20260620 22:52:11.453403 5920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:33825 every 8 connection(s)
I20260620 22:52:11.453652 5792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:11.458346 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5792
I20260620 22:52:11.458446 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:11.458618 2410 external_mini_cluster.cc:1428] Setting key 5850d311e61b611ef122ce8a516437c8
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:11.462046 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.454283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:41663 (local address 127.2.90.190:42449)
0620 22:52:11.454321 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:11.454322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:11.454548 (+ 226us) server_negotiation.cc:407] Connection header received
0620 22:52:11.455063 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:11.455065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:11.455076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:11.455125 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:11.456007 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.456499 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.457474 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.457656 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.457704 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.459833 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:11.459834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:11.459834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:11.459841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:11.461079 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.461390 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.461390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.461391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.461443 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.461612 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.461612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.461613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.461700 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:11.461742 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:11.461934 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 22:52:11.461985 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:11.462208 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.454268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42449 (local address 127.2.90.131:41663)
0620 22:52:11.454434 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:11.454469 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:11.454972 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:11.455192 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:11.455196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:11.455726 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:11.455909 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:11.455913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.456568 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.456569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:11.457403 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:11.457406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.457717 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.457718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:11.457814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.458267 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:11.458277 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:11.459759 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:11.461139 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:11.461142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:11.461144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:11.461336 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:11.461487 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:11.461488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:11.461488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:11.461571 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:11.461746 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:11.461748 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:11.461849 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 22:52:11.461921 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:52:11.462026 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260620 22:52:11.462567 5921 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42449
I20260620 22:52:11.462653 5921 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:11.462812 5921 heartbeater.cc:511] Master 127.2.90.190:42449 requested a full tablet report, sending...
I20260620 22:52:11.463143 5461 ts_manager.cc:194] Registered new tserver with Master: e0ba1ee5ae6d41b99c5fb02746dedba8 (127.2.90.131:33825)
I20260620 22:52:11.463470 5461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:41663
I20260620 22:52:11.470105 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995931, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:11.490067 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:11.483753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.2.90.190:42449)
0620 22:52:11.483813 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:11.483815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:11.483858 (+ 43us) server_negotiation.cc:407] Connection header received
0620 22:52:11.483939 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:11.483940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:11.483950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:11.483995 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:11.484504 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.485019 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.485714 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:11.485895 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:11.485945 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:11.487891 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:11.487892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:11.487893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:11.487899 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:11.489084 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.489372 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.489373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.489374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.489429 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:11.489591 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:11.489592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:11.489592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:11.489726 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:11.489811 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:11.489908 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 22:52:11.489982 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:52:11.490269 5461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54426
I20260620 22:52:11.490665 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5524
I20260620 22:52:11.494359 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5658
I20260620 22:52:11.497660 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5792
I20260620 22:52:11.501195 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5430
2026-06-20T22:52:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1736 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5933](info): set up 2 sockets
Jun 20 22:52:11 dist-test-slave-4mtw krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260620 22:52:12.456321 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:12 dist-test-slave-4mtw krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:12.561475 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35653
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:12.638824 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:12.639014 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:12.639034 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:12.640121 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:12.640154 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:12.640168 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:12.640178 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:12.640189 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:12.641716 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:12.641908 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:12.642077 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:12.646631 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:12.646718 5946 server_base.cc:1034] running on GCE node
W20260620 22:52:12.646643 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:12.646643 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:12.647084 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:12.647341 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:12.648478 5946 hybrid_clock.cc:648] HybridClock initialized: now 1781995932648466 us; error 30 us; skew 500 ppm
Jun 20 22:52:12 dist-test-slave-4mtw krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:12.650791 5946 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:12.651225 5946 webserver.cc:460] Webserver started at http://127.2.90.190:33135/ using document root <none> and password file <none>
I20260620 22:52:12.651441 5946 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:12.651489 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:12.651619 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:12.652639 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "55bf9fcc1bc64d18af04a2b49439e4db"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "88c0d3ac8ec89e7f33857236dd934c9f"
server_key_iv: "623a69136e8cee21aa80d1f7712b4b77"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.653010 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "55bf9fcc1bc64d18af04a2b49439e4db"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "88c0d3ac8ec89e7f33857236dd934c9f"
server_key_iv: "623a69136e8cee21aa80d1f7712b4b77"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.654211 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:12.654927 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:12.655071 5946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:12.655153 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "55bf9fcc1bc64d18af04a2b49439e4db"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "88c0d3ac8ec89e7f33857236dd934c9f"
server_key_iv: "623a69136e8cee21aa80d1f7712b4b77"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.655220 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:12.661440 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:12.663321 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:12.663455 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:12.667174 5946 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35653
I20260620 22:52:12.667204 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35653 every 8 connection(s)
I20260620 22:52:12.667634 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:12.668262 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:12.670457 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db: Bootstrap starting.
I20260620 22:52:12.671003 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:12.671257 6014 log.cc:824] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:12.671865 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db: No bootstrap required, opened a new log
I20260620 22:52:12.673461 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35653 } }
I20260620 22:52:12.673590 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:12.673611 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55bf9fcc1bc64d18af04a2b49439e4db, State: Initialized, Role: FOLLOWER
I20260620 22:52:12.673749 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35653 } }
I20260620 22:52:12.673822 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:12.673854 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:12.673899 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:12.674716 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35653 } }
I20260620 22:52:12.674811 6014 leader_election.cc:302] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55bf9fcc1bc64d18af04a2b49439e4db; no voters:
I20260620 22:52:12.674961 6014 leader_election.cc:288] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:12.675000 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:12.675155 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [term 1 LEADER]: Becoming Leader. State: Replica: 55bf9fcc1bc64d18af04a2b49439e4db, State: Running, Role: LEADER
I20260620 22:52:12.675242 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35653 } }
I20260620 22:52:12.675282 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 5946
I20260620 22:52:12.675289 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:12.675374 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:12.675554 2410 external_mini_cluster.cc:1428] Setting key a2eaf986a4e2b45519af581cf7b966b5
I20260620 22:52:12.675561 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35653 } } }
I20260620 22:52:12.675581 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55bf9fcc1bc64d18af04a2b49439e4db. Latest consensus state: current_term: 1 leader_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bf9fcc1bc64d18af04a2b49439e4db" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35653 } } }
I20260620 22:52:12.675627 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:12.675642 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:12.676229 6022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:12.676812 6022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:12.678519 6022 catalog_manager.cc:1347] Generated new cluster ID: 8faf04d19d404cf497be458ce64af8a4
I20260620 22:52:12.678571 6022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:52:12 dist-test-slave-4mtw krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:12.685200 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:12.675971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.2.90.190:35653)
0620 22:52:12.676142 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:12.676150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:12.676949 (+ 799us) server_negotiation.cc:407] Connection header received
0620 22:52:12.677082 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:12.677090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:12.677156 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:12.677275 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:12.677979 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:12.679044 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:12.679757 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:12.680174 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:12.680269 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:12.682389 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:12.682393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:12.682393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:12.682405 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:12.683847 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:12.684159 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:12.684163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:12.684163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:12.684248 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:12.684423 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:12.684424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:12.684424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:12.684633 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:12.684807 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:12.684898 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:52:12.685024 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260620 22:52:12.695324 6022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:12.695967 6022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:12.705089 6022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55bf9fcc1bc64d18af04a2b49439e4db: Generated new TSK 0
I20260620 22:52:12.705253 6022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:12.733665 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35653
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:12.833405 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:12.833598 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:12.833619 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:12.834754 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:12.834796 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:12.834820 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:12.836305 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:12.836515 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:12.836755 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:12.841455 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:12.841452 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:12.841470 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:12.841755 6040 server_base.cc:1034] running on GCE node
I20260620 22:52:12.841977 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:12.842196 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:12.843334 6040 hybrid_clock.cc:648] HybridClock initialized: now 1781995932843320 us; error 29 us; skew 500 ppm
Jun 20 22:52:12 dist-test-slave-4mtw krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:12.845641 6040 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:12.846089 6040 webserver.cc:460] Webserver started at http://127.2.90.129:39413/ using document root <none> and password file <none>
I20260620 22:52:12.846282 6040 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:12.846328 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:12.846446 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:12.847471 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "b298f0460c8b48e79968e4bb16a42517"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "bcad9e1f5d3d27ebcdecd42c27015e1f"
server_key_iv: "36e1e8c20fdfff9b41dea3cca7e0c906"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.847785 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "b298f0460c8b48e79968e4bb16a42517"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "bcad9e1f5d3d27ebcdecd42c27015e1f"
server_key_iv: "36e1e8c20fdfff9b41dea3cca7e0c906"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.849076 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:12.849731 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:12.849897 6040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:12.849980 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "b298f0460c8b48e79968e4bb16a42517"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "bcad9e1f5d3d27ebcdecd42c27015e1f"
server_key_iv: "36e1e8c20fdfff9b41dea3cca7e0c906"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.850037 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:12.867072 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:12.868753 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:12.868883 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:12.869096 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:12.869377 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:12.869410 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:12.869437 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:12.869453 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:12.874774 6040 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37953
I20260620 22:52:12.874835 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37953 every 8 connection(s)
I20260620 22:52:12.875130 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:12.878676 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6040
I20260620 22:52:12.878749 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:12.878909 2410 external_mini_cluster.cc:1428] Setting key 9687b43577170dc1e7c6fe060d2b7435
Jun 20 22:52:12 dist-test-slave-4mtw krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:12.884291 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:12.875760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:43775 (local address 127.2.90.190:35653)
0620 22:52:12.875800 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:12.875802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:12.876028 (+ 226us) server_negotiation.cc:407] Connection header received
0620 22:52:12.877036 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:12.877038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:12.877050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:12.877091 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:12.877709 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:12.878215 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:12.879312 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:12.879566 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:12.879621 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:12.881763 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:12.881764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:12.881765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:12.881775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:12.883067 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:12.883419 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:12.883420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:12.883420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:12.883475 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:12.883691 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:12.883692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:12.883692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:12.883804 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:12.883859 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:12.884118 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 22:52:12.884204 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:52:12.884455 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:12.875769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35653 (local address 127.2.90.129:43775)
0620 22:52:12.875908 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:12.875945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:12.876943 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:12.877170 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:12.877174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:12.877433 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:12.877614 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:12.877619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:12.878299 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:12.878299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:12.879244 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:12.879246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:12.879639 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:12.879640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:12.879764 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:12.880219 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:12.880229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:12.881676 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:12.883142 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:12.883149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:12.883151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:12.883351 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:12.883531 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:12.883531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:12.883532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:12.883634 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:12.883861 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:12.883867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:12.883998 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 22:52:12.884085 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 22:52:12.884231 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260620 22:52:12.885067 6169 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35653
I20260620 22:52:12.885149 6169 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:12.885310 6169 heartbeater.cc:511] Master 127.2.90.190:35653 requested a full tablet report, sending...
I20260620 22:52:12.885794 5978 ts_manager.cc:194] Registered new tserver with Master: b298f0460c8b48e79968e4bb16a42517 (127.2.90.129:37953)
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:52:12.886363 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:43775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:12.901240 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35653
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:12.978147 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:12.978359 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:12.978385 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:12.979549 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:12.979588 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:12.979615 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:12.981110 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:12.981328 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:12.981541 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:12.986234 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:12.986266 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:12.986403 6174 server_base.cc:1034] running on GCE node
W20260620 22:52:12.986286 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:12.986598 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:12.986829 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:12.987967 6174 hybrid_clock.cc:648] HybridClock initialized: now 1781995932987945 us; error 33 us; skew 500 ppm
Jun 20 22:52:12 dist-test-slave-4mtw krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:12.990249 6174 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:12.990658 6174 webserver.cc:460] Webserver started at http://127.2.90.130:45907/ using document root <none> and password file <none>
I20260620 22:52:12.990844 6174 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:12.990890 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:12.991019 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:12.992020 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "8ae5c2fe71984b99a3dca243cb8a8d7a"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "b26e789c8d7b0ea13e2fb12aa87c2eb6"
server_key_iv: "faaaa36af4c23b1c09996e360995ce83"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.992349 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "8ae5c2fe71984b99a3dca243cb8a8d7a"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "b26e789c8d7b0ea13e2fb12aa87c2eb6"
server_key_iv: "faaaa36af4c23b1c09996e360995ce83"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.993613 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:52:12.994318 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:12.994477 6174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:12.994563 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "8ae5c2fe71984b99a3dca243cb8a8d7a"
format_stamp: "Formatted at 2026-06-20 22:52:12 on dist-test-slave-4mtw"
server_key: "b26e789c8d7b0ea13e2fb12aa87c2eb6"
server_key_iv: "faaaa36af4c23b1c09996e360995ce83"
server_key_version: "encryptionkey@0"
I20260620 22:52:12.994622 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:13.011080 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:13.012898 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:13.013047 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:13.013258 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:13.013541 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:13.013573 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:13.013602 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:13.013619 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:13.019014 6174 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:36479
I20260620 22:52:13.019061 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:36479 every 8 connection(s)
I20260620 22:52:13.019358 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995932, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:13.025267 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6174
I20260620 22:52:13.025364 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:13.025519 2410 external_mini_cluster.cc:1428] Setting key 984452b6a751248b14059b008256049c
I20260620 22:52:13.027971 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:13.020017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:42219 (local address 127.2.90.190:35653)
0620 22:52:13.020069 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:13.020071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:13.020260 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:52:13.021287 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:13.021288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:13.021297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:13.021348 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:13.021894 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.022375 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.023293 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.023466 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.023508 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:13.025359 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:13.025361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:13.025361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:13.025369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:13.026659 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.027027 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.027028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.027028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.027085 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.027388 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.027388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.027389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.027522 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:13.027584 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:13.027798 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 22:52:13.027883 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:52:13.028048 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:13.019999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35653 (local address 127.2.90.130:42219)
0620 22:52:13.020141 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:13.020181 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:13.021200 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:13.021405 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:13.021408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:13.021640 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:13.021821 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:13.021825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.022439 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.022440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:13.023232 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:13.023234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.023516 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.023517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:13.023605 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:13.023992 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:13.023999 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:13.025258 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:13.026734 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:13.026739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:13.026741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:13.026956 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:13.027173 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:13.027174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:13.027175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:13.027325 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:13.027578 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:13.027581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:13.027690 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 22:52:13.027766 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:52:13.027883 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260620 22:52:13.028440 6303 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35653
I20260620 22:52:13.028525 6303 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:13.028713 6303 heartbeater.cc:511] Master 127.2.90.190:35653 requested a full tablet report, sending...
I20260620 22:52:13.029083 5978 ts_manager.cc:194] Registered new tserver with Master: 8ae5c2fe71984b99a3dca243cb8a8d7a (127.2.90.130:36479)
I20260620 22:52:13.029438 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:42219
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:13.047291 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35653
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:13.124003 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:13.124203 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:13.124233 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:13.125375 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:13.125413 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:13.125447 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:13.126926 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:13.127157 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:13.127358 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:13.131807 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:13.131803 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:13.131803 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:13.131882 6308 server_base.cc:1034] running on GCE node
I20260620 22:52:13.132185 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:13.132402 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:13.133544 6308 hybrid_clock.cc:648] HybridClock initialized: now 1781995933133530 us; error 27 us; skew 500 ppm
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995933, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:13.135811 6308 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:13.136233 6308 webserver.cc:460] Webserver started at http://127.2.90.131:35901/ using document root <none> and password file <none>
I20260620 22:52:13.136430 6308 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:13.136475 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:13.136593 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:13.137689 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "d24cc0a8a64349f286942236a6c64e71"
format_stamp: "Formatted at 2026-06-20 22:52:13 on dist-test-slave-4mtw"
server_key: "410a42df97a1ebb39d60ba9c256df258"
server_key_iv: "54689b17008ed70bb5276f80af962508"
server_key_version: "encryptionkey@0"
I20260620 22:52:13.138006 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "d24cc0a8a64349f286942236a6c64e71"
format_stamp: "Formatted at 2026-06-20 22:52:13 on dist-test-slave-4mtw"
server_key: "410a42df97a1ebb39d60ba9c256df258"
server_key_iv: "54689b17008ed70bb5276f80af962508"
server_key_version: "encryptionkey@0"
I20260620 22:52:13.139160 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:13.139814 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:13.139995 6308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:13.140082 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d24cc0a8a64349f286942236a6c64e71"
format_stamp: "Formatted at 2026-06-20 22:52:13 on dist-test-slave-4mtw"
server_key: "410a42df97a1ebb39d60ba9c256df258"
server_key_iv: "54689b17008ed70bb5276f80af962508"
server_key_version: "encryptionkey@0"
I20260620 22:52:13.140146 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:13.157286 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:13.159053 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:13.159189 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:13.159400 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:13.159708 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:13.159742 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:13.159771 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:13.159798 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:13.166971 6308 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:36747
I20260620 22:52:13.167040 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:36747 every 8 connection(s)
I20260620 22:52:13.167320 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:13.171540 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6308
I20260620 22:52:13.171625 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:13.171789 2410 external_mini_cluster.cc:1428] Setting key 6b2068f5bd8bc199b74a90b60f47d872
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995933, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:13.176241 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:13.168054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:40785 (local address 127.2.90.190:35653)
0620 22:52:13.168117 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:13.168119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:13.168304 (+ 185us) server_negotiation.cc:407] Connection header received
0620 22:52:13.169277 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:13.169278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:13.169291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:13.169342 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:13.169939 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.170438 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.171422 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.171623 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.171683 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:13.173874 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:13.173876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:13.173876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:13.173883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:13.174965 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.175365 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.175367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.175367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.175435 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.175673 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.175674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.175675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.175800 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:13.175858 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:13.176098 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 22:52:13.176178 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:52:13.176453 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:13.168023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35653 (local address 127.2.90.131:40785)
0620 22:52:13.168189 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:13.168223 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:13.169176 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:13.169413 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:13.169416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:13.169665 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:13.169849 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:13.169854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.170522 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.170522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:13.171310 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:13.171312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.171747 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.171748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:13.171891 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:13.172422 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:13.172433 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:13.173791 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:13.175052 (+ 1261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:13.175058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:13.175060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:13.175270 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:13.175492 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:13.175493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:13.175493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:13.175610 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:13.175866 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:13.175872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:13.175993 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 22:52:13.176077 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:52:13.176219 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260620 22:52:13.176916 6437 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35653
I20260620 22:52:13.177003 6437 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:13.177167 6437 heartbeater.cc:511] Master 127.2.90.190:35653 requested a full tablet report, sending...
I20260620 22:52:13.177503 5978 ts_manager.cc:194] Registered new tserver with Master: d24cc0a8a64349f286942236a6c64e71 (127.2.90.131:36747)
I20260620 22:52:13.177855 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:40785
I20260620 22:52:13.183355 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:52:13.203840 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:13.197155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.2.90.129:37953)
0620 22:52:13.197325 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:13.197330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:13.197342 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:52:13.197366 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:13.197367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:13.197381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:13.197450 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:13.197931 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.198666 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.199348 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.199648 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.199695 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:13.201683 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:13.201691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:13.201691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:13.201698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:13.202963 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.203231 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.203234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.203234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.203293 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.203411 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.203411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.203412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.203518 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:13.203618 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:13.203681 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 22:52:13.203739 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:13.210688 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:13.205034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.2.90.190:35653)
0620 22:52:13.205076 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:13.205078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:13.205087 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:52:13.205111 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:13.205112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:13.205120 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:13.205161 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:13.205610 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.206073 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.206684 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:13.206856 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:13.206901 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:13.208760 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:13.208761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:13.208762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:13.208768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:13.209911 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.210151 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.210152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.210152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.210192 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:13.210308 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:13.210308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:13.210308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:13.210437 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:13.210502 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:13.210570 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 22:52:13.210622 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:52:13.210932 5978 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578
I20260620 22:52:13.211223 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6040
I20260620 22:52:13.215405 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6174
I20260620 22:52:13.219045 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6308
I20260620 22:52:13.222448 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 5946
2026-06-20T22:52:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1721 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[6449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[6449](info): set up 2 sockets
Jun 20 22:52:13 dist-test-slave-4mtw krb5kdc[6449](info): commencing operation
krb5kdc: starting...
W20260620 22:52:14.160255 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:14.266609 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45575
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:14.343354 6462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:14.343559 6462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:14.343588 6462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:14.344732 6462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:14.344771 6462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:14.344789 6462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:14.344805 6462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:14.344820 6462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:14.346263 6462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:45575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:14.346505 6462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:14.346695 6462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:14.351065 6470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:14.351075 6467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:14.351075 6468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:14.351202 6462 server_base.cc:1034] running on GCE node
I20260620 22:52:14.351521 6462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:14.351794 6462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:14.352931 6462 hybrid_clock.cc:648] HybridClock initialized: now 1781995934352907 us; error 35 us; skew 500 ppm
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:14.355199 6462 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:14.355477 6462 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:14.355517 6462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:14.355603 6462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:14.356540 6462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "44329ca1df014aeba8be9bd3446570af"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "e71101ed4849a6690ca54f6542fed952"
server_key_iv: "28dc1fd26c54234f984c71501295de08"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.356863 6462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "44329ca1df014aeba8be9bd3446570af"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "e71101ed4849a6690ca54f6542fed952"
server_key_iv: "28dc1fd26c54234f984c71501295de08"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.357940 6462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:14.358525 6476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.358662 6462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:14.358728 6462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "44329ca1df014aeba8be9bd3446570af"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "e71101ed4849a6690ca54f6542fed952"
server_key_iv: "28dc1fd26c54234f984c71501295de08"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.358794 6462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:14.392800 6462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:14.394510 6462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:14.394631 6462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:14.398192 6462 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:45575
I20260620 22:52:14.398219 6528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:45575 every 8 connection(s)
I20260620 22:52:14.398481 6462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:14.399322 6529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:14.400897 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6462
I20260620 22:52:14.401008 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:14.401177 2410 external_mini_cluster.cc:1428] Setting key cd3b2bc762638c43268f654f68d4f378
I20260620 22:52:14.401825 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af: Bootstrap starting.
I20260620 22:52:14.402384 6529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:14.402693 6529 log.cc:824] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:14.403362 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af: No bootstrap required, opened a new log
I20260620 22:52:14.405359 6529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44329ca1df014aeba8be9bd3446570af" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45575 } }
I20260620 22:52:14.405498 6529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:14.405532 6529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44329ca1df014aeba8be9bd3446570af, State: Initialized, Role: FOLLOWER
I20260620 22:52:14.405694 6529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44329ca1df014aeba8be9bd3446570af" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45575 } }
I20260620 22:52:14.405781 6529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:14.405807 6529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:14.405853 6529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:14.406342 6529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44329ca1df014aeba8be9bd3446570af" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45575 } }
I20260620 22:52:14.406427 6529 leader_election.cc:302] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44329ca1df014aeba8be9bd3446570af; no voters:
I20260620 22:52:14.406569 6529 leader_election.cc:288] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:14.406615 6534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:14.406740 6534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [term 1 LEADER]: Becoming Leader. State: Replica: 44329ca1df014aeba8be9bd3446570af, State: Running, Role: LEADER
I20260620 22:52:14.406839 6529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:14.406867 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44329ca1df014aeba8be9bd3446570af" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45575 } }
I20260620 22:52:14.407178 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44329ca1df014aeba8be9bd3446570af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44329ca1df014aeba8be9bd3446570af" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45575 } } }
I20260620 22:52:14.407263 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:14.407166 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44329ca1df014aeba8be9bd3446570af. Latest consensus state: current_term: 1 leader_uuid: "44329ca1df014aeba8be9bd3446570af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44329ca1df014aeba8be9bd3446570af" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45575 } } }
I20260620 22:52:14.407452 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af [sys.catalog]: This master's current role is: LEADER
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:14.408191 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:14.409008 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:14.410386 6541 catalog_manager.cc:1347] Generated new cluster ID: 1cd4b7b770794a638490832c20b99c54
I20260620 22:52:14.410446 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:14.411322 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.401658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.2.90.190:45575)
0620 22:52:14.401797 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.401805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.401830 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:52:14.402256 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.402262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.402315 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.402432 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:14.403130 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.403994 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.405053 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.405439 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.405521 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.408001 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:14.408005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:14.408006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:14.408018 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:14.409859 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.410229 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.410232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.410232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.410334 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.410544 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.410545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.410545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.410759 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:14.410901 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.411010 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.411140 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260620 22:52:14.426011 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:14.426554 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:14.431485 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af: Generated new TSK 0
I20260620 22:52:14.431633 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:14.455767 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45575
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:14.532666 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:14.532871 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:14.532891 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:14.534016 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:14.534057 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:14.534082 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:14.535542 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:14.535758 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:14.535959 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:14.540380 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:14.540387 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:14.540391 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:14.540577 6555 server_base.cc:1034] running on GCE node
I20260620 22:52:14.540777 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:14.540987 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:14.542133 6555 hybrid_clock.cc:648] HybridClock initialized: now 1781995934542121 us; error 27 us; skew 500 ppm
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:14.544351 6555 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:14.544716 6555 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:14.544776 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:14.544898 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:14.545853 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "b3000d836f8b4d6c9af444c901e08bb3"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "42a63d9f7fee89c1a8b6abdbca97cf18"
server_key_iv: "a0cec5da4953bee3965ea9efedae5a8c"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.546161 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "b3000d836f8b4d6c9af444c901e08bb3"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "42a63d9f7fee89c1a8b6abdbca97cf18"
server_key_iv: "a0cec5da4953bee3965ea9efedae5a8c"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.547266 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:14.547945 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.548098 6555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:14.548161 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "b3000d836f8b4d6c9af444c901e08bb3"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "42a63d9f7fee89c1a8b6abdbca97cf18"
server_key_iv: "a0cec5da4953bee3965ea9efedae5a8c"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.548224 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:14.555014 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:14.556993 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:14.557117 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:14.557312 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:14.557593 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:14.557626 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.557646 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:14.557668 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.563094 6555 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:36533
I20260620 22:52:14.563125 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:36533 every 8 connection(s)
I20260620 22:52:14.563362 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:14.570259 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6555
I20260620 22:52:14.570341 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:14.570494 2410 external_mini_cluster.cc:1428] Setting key 688c17b555c4a3eb829c81f1e0bde532
I20260620 22:52:14.571907 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.563978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:38725 (local address 127.2.90.190:45575)
0620 22:52:14.564034 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.564036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.564237 (+ 201us) server_negotiation.cc:407] Connection header received
0620 22:52:14.564792 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.564793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.564804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.564853 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:14.565731 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.566231 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.567195 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.567409 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.567465 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.569460 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:14.569462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:14.569462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:14.569469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:14.570627 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.571026 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.571027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.571027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.571087 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.571319 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.571320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.571320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.571437 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:14.571493 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.571742 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.571823 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:14.572072 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.563970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45575 (local address 127.2.90.129:38725)
0620 22:52:14.564129 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:14.564162 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:14.564703 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:14.564925 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:14.564929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:14.565163 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:14.565634 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.565639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.566315 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.566316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.567108 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.567110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.567479 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.567480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.567605 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.568074 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:14.568085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:14.569373 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:14.570699 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:14.570705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:14.570707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:14.570952 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:14.571162 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:14.571162 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:14.571163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:14.571257 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:14.571494 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:14.571500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:14.571629 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 22:52:14.571712 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:52:14.571854 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260620 22:52:14.572507 6683 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45575
I20260620 22:52:14.572589 6683 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:14.572752 6683 heartbeater.cc:511] Master 127.2.90.190:45575 requested a full tablet report, sending...
I20260620 22:52:14.573292 6493 ts_manager.cc:194] Registered new tserver with Master: b3000d836f8b4d6c9af444c901e08bb3 (127.2.90.129:36533)
I20260620 22:52:14.574122 6493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:38725
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:14.593375 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45575
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:14.672137 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:14.672344 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:14.672367 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:14.673529 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:14.673571 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:14.673597 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:14.675034 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:14.675252 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:14.675482 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:14.680155 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:14.680160 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:14.680330 6688 server_base.cc:1034] running on GCE node
W20260620 22:52:14.680425 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:14.680614 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:14.680879 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:14.682013 6688 hybrid_clock.cc:648] HybridClock initialized: now 1781995934682001 us; error 28 us; skew 500 ppm
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:14.684240 6688 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:14.684556 6688 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:14.684604 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:14.684748 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:14.685745 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "a753a8ccdcb5405cbad8a55b9742165c"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "bc4f1101c0e9824d109d5b3ee0282abb"
server_key_iv: "d3af3416673ef21a07470952bdd22cc5"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.686074 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "a753a8ccdcb5405cbad8a55b9742165c"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "bc4f1101c0e9824d109d5b3ee0282abb"
server_key_iv: "d3af3416673ef21a07470952bdd22cc5"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.687222 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:14.687853 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.688007 6688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.688078 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "a753a8ccdcb5405cbad8a55b9742165c"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "bc4f1101c0e9824d109d5b3ee0282abb"
server_key_iv: "d3af3416673ef21a07470952bdd22cc5"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.688138 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:14.734547 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:14.736246 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:14.736384 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:14.736617 6688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:14.736954 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:14.736989 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.737017 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:14.737035 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.742238 6688 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45109
I20260620 22:52:14.742287 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45109 every 8 connection(s)
I20260620 22:52:14.742534 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:14.748234 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6688
I20260620 22:52:14.748324 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:14.748485 2410 external_mini_cluster.cc:1428] Setting key 96653b2beac3a8673ab77114ca020091
I20260620 22:52:14.751149 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.743160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:45485 (local address 127.2.90.190:45575)
0620 22:52:14.743200 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.743202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.743415 (+ 213us) server_negotiation.cc:407] Connection header received
0620 22:52:14.743890 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.743891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.743903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.743953 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:14.744907 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.745384 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.746360 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.746554 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.746613 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.748609 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:14.748611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:14.748611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:14.748618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:14.749800 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.750169 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.750171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.750171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.750244 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.750500 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.750500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.750501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.750639 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:14.750695 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.751019 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.751086 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:52:14.751371 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.743143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45575 (local address 127.2.90.130:45485)
0620 22:52:14.743292 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:14.743328 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:14.743798 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:14.744025 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:14.744029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:14.744283 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:14.744820 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.744826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.745471 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.745472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.746275 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.746277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.746631 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.746632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.746760 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.747163 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:14.747171 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:14.748532 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:14.749872 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:14.749879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:14.749881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:14.750087 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:14.750314 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:14.750315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:14.750316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:14.750442 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:14.750703 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:14.750708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:14.750883 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 22:52:14.751000 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 22:52:14.751160 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260620 22:52:14.751844 6816 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45575
I20260620 22:52:14.751943 6816 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:14.752086 6816 heartbeater.cc:511] Master 127.2.90.190:45575 requested a full tablet report, sending...
I20260620 22:52:14.752462 6493 ts_manager.cc:194] Registered new tserver with Master: a753a8ccdcb5405cbad8a55b9742165c (127.2.90.130:45109)
I20260620 22:52:14.752871 6493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:45485
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:14.769724 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45575
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:14.845701 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:14.845890 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:14.845911 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:14.847031 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:14.847069 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:14.847093 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:14.848521 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:14.848732 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:14.848927 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:14.853407 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:14.853407 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:14.853443 6821 server_base.cc:1034] running on GCE node
W20260620 22:52:14.853446 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:14.853793 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:14.853992 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:14.855120 6821 hybrid_clock.cc:648] HybridClock initialized: now 1781995934855112 us; error 26 us; skew 500 ppm
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:14.857394 6821 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:14.857715 6821 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:14.857764 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:14.857885 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:14.858985 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "59dce15d7ffa4d0c842e00e109c5b35b"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "0844a42782b23ea3dd767a21ea3ac3a0"
server_key_iv: "187fd74e481db7aef7b5390317178435"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.859347 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "59dce15d7ffa4d0c842e00e109c5b35b"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "0844a42782b23ea3dd767a21ea3ac3a0"
server_key_iv: "187fd74e481db7aef7b5390317178435"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.860536 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:14.861253 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.861426 6821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.861507 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "59dce15d7ffa4d0c842e00e109c5b35b"
format_stamp: "Formatted at 2026-06-20 22:52:14 on dist-test-slave-4mtw"
server_key: "0844a42782b23ea3dd767a21ea3ac3a0"
server_key_iv: "187fd74e481db7aef7b5390317178435"
server_key_version: "encryptionkey@0"
I20260620 22:52:14.861572 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:14.870368 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:14.871922 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:14.872046 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:14.872248 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:14.872512 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:14.872543 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.872570 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:14.872587 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:14.878865 6821 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42199
I20260620 22:52:14.878897 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42199 every 8 connection(s)
I20260620 22:52:14.879173 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:14.883388 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6821
I20260620 22:52:14.883486 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:14.883687 2410 external_mini_cluster.cc:1428] Setting key 226e8e0da8981489f75c500bc010e98a
Jun 20 22:52:14 dist-test-slave-4mtw krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995934, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:14.888141 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.879873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:50569 (local address 127.2.90.190:45575)
0620 22:52:14.879921 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.879923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.880141 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:52:14.880793 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.880794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.880803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.880843 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:14.881865 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.882501 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.883698 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.883934 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.883993 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.886020 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:14.886021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:14.886022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:14.886027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:14.887118 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.887461 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.887462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.887462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.887507 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.887666 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.887666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.887666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.887755 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:14.887800 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.888025 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.888081 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:52:14.888319 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.879835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45575 (local address 127.2.90.131:50569)
0620 22:52:14.879987 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:14.880052 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:14.880667 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:14.880914 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:14.880918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:14.881242 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:14.881779 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.881785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.882580 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.882581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.883619 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.883622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.884020 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.884021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.884157 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.884655 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:14.884663 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:14.885947 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:14.887181 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:14.887186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:14.887188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:14.887403 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:14.887545 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:14.887545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:14.887545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:14.887626 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:14.887823 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:14.887826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:14.887929 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 22:52:14.888005 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:52:14.888113 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260620 22:52:14.888736 6949 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45575
I20260620 22:52:14.888820 6949 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:14.888928 6949 heartbeater.cc:511] Master 127.2.90.190:45575 requested a full tablet report, sending...
I20260620 22:52:14.889276 6493 ts_manager.cc:194] Registered new tserver with Master: 59dce15d7ffa4d0c842e00e109c5b35b (127.2.90.131:42199)
I20260620 22:52:14.889627 6493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:50569
I20260620 22:52:14.895212 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:52:14.901280 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.896395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.2.90.190:45575)
0620 22:52:14.896457 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.896459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.896528 (+ 69us) server_negotiation.cc:407] Connection header received
0620 22:52:14.896566 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.896567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.896574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.896616 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:14.897180 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.897633 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.898275 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.898442 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.898481 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.899415 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:14.899416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:14.899416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:14.899423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:14.900417 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.900722 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.900723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.900724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.900773 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:14.900897 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:14.900897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:14.900897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:14.901026 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:14.901094 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.901170 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.901220 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:52:14.902731 6493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:14.904225 6493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:14.910288 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.906351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36533 (local address 127.0.0.1:56674)
0620 22:52:14.906661 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:14.906669 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:14.906706 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:14.906997 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:14.906997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:14.906999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:14.907049 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.907050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.908313 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.908314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.909271 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.909273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.910015 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.910016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.910089 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.910105 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 22:52:14.910173 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:52:14.910206 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":39,"threads_started":1}
I20260620 22:52:14.910311 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.906530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.2.90.129:36533)
0620 22:52:14.906803 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.906807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.906815 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:14.906848 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.906850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.906862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.906923 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:14.907117 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.908213 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.909343 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.909933 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.910002 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.910059 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.910193 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.910232 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":65,"threads_started":1}
I20260620 22:52:14.910465 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.906260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.2.90.130:45109)
0620 22:52:14.906458 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.906462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.906472 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:14.906507 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.906509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.906525 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.906619 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:14.906793 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.907954 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.909444 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.909989 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.910247 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.910308 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.910332 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.910371 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260620 22:52:14.910554 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.906154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45109 (local address 127.0.0.1:32874)
0620 22:52:14.906315 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:14.906349 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:14.906438 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:14.906625 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:14.906628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:14.906630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:14.906696 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.906697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.908038 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.908039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.909348 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.909351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.910078 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.910079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.910125 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.910128 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:14.910153 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 22:52:14.910206 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260620 22:52:14.910708 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.906452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.2.90.131:42199)
0620 22:52:14.906584 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:14.906588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:14.906753 (+ 165us) server_negotiation.cc:407] Connection header received
0620 22:52:14.906794 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:14.906797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:14.906811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:14.906889 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:14.907055 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.908256 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.909632 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.910251 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.910312 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.910363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:14.910590 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 22:52:14.910628 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260620 22:52:14.910851 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:14.906186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42199 (local address 127.0.0.1:55088)
0620 22:52:14.906609 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:14.906618 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:14.906674 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:14.906901 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:14.906902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:14.906904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:14.906939 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.906941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.908344 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.908345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.909559 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:14.909561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:14.910395 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:14.910396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:14.910470 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:14.910474 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:14.910517 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:52:14.910553 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":36,"thread_start_us":58,"threads_started":1}
I20260620 22:52:14.911304 6750 tablet_service.cc:1467] Processing CreateTablet for tablet 4ededfcf199f400bbe29be82567eadaa (DEFAULT_TABLE table=test-table [id=d4ac1676d51d4ae9974cae7585743078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:14.911316 6617 tablet_service.cc:1467] Processing CreateTablet for tablet 4ededfcf199f400bbe29be82567eadaa (DEFAULT_TABLE table=test-table [id=d4ac1676d51d4ae9974cae7585743078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:14.911453 6883 tablet_service.cc:1467] Processing CreateTablet for tablet 4ededfcf199f400bbe29be82567eadaa (DEFAULT_TABLE table=test-table [id=d4ac1676d51d4ae9974cae7585743078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:14.911603 6750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ededfcf199f400bbe29be82567eadaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:14.911625 6617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ededfcf199f400bbe29be82567eadaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:14.911695 6883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ededfcf199f400bbe29be82567eadaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:14.914031 6963 tablet_bootstrap.cc:492] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Bootstrap starting.
I20260620 22:52:14.914649 6963 tablet_bootstrap.cc:654] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:14.914747 6965 tablet_bootstrap.cc:492] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: Bootstrap starting.
I20260620 22:52:14.915220 6963 log.cc:824] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:14.915410 6965 tablet_bootstrap.cc:654] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:14.915946 6965 log.cc:824] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:14.915977 6963 tablet_bootstrap.cc:492] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: No bootstrap required, opened a new log
I20260620 22:52:14.916035 6963 ts_tablet_manager.cc:1397] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:14.916254 6964 tablet_bootstrap.cc:492] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Bootstrap starting.
I20260620 22:52:14.916545 6965 tablet_bootstrap.cc:492] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: No bootstrap required, opened a new log
I20260620 22:52:14.916600 6965 ts_tablet_manager.cc:1397] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:14.916846 6964 tablet_bootstrap.cc:654] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:14.917387 6964 log.cc:824] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:14.918051 6964 tablet_bootstrap.cc:492] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: No bootstrap required, opened a new log
I20260620 22:52:14.918111 6964 ts_tablet_manager.cc:1397] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:14.918012 6963 raft_consensus.cc:348] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:14.918164 6963 raft_consensus.cc:374] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:14.918205 6963 raft_consensus.cc:729] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a753a8ccdcb5405cbad8a55b9742165c, State: Initialized, Role: FOLLOWER
I20260620 22:52:14.918144 6965 raft_consensus.cc:348] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:14.918298 6965 raft_consensus.cc:374] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:14.918332 6965 raft_consensus.cc:729] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3000d836f8b4d6c9af444c901e08bb3, State: Initialized, Role: FOLLOWER
I20260620 22:52:14.918320 6963 consensus_queue.cc:260] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:14.918447 6965 consensus_queue.cc:260] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:14.918583 6963 ts_tablet_manager.cc:1428] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:52:14.918694 6816 heartbeater.cc:503] Master 127.2.90.190:45575 was elected leader, sending a full tablet report...
I20260620 22:52:14.918702 6965 ts_tablet_manager.cc:1428] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:52:14.918824 6683 heartbeater.cc:503] Master 127.2.90.190:45575 was elected leader, sending a full tablet report...
I20260620 22:52:14.919791 6964 raft_consensus.cc:348] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:14.919915 6964 raft_consensus.cc:374] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:14.919942 6964 raft_consensus.cc:729] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59dce15d7ffa4d0c842e00e109c5b35b, State: Initialized, Role: FOLLOWER
I20260620 22:52:14.920018 6964 consensus_queue.cc:260] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:14.920226 6964 ts_tablet_manager.cc:1428] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:14.920265 6949 heartbeater.cc:503] Master 127.2.90.190:45575 was elected leader, sending a full tablet report...
W20260620 22:52:14.993003 6817 tablet.cc:2369] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:52:15.063970 6684 tablet.cc:2369] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:52:15.129758 6950 tablet.cc:2369] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:15.131717 6970 raft_consensus.cc:482] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:15.131819 6970 raft_consensus.cc:504] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:15.132179 6970 leader_election.cc:288] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a753a8ccdcb5405cbad8a55b9742165c (127.2.90.130:45109), 59dce15d7ffa4d0c842e00e109c5b35b (127.2.90.131:42199)
I20260620 22:52:15.136029 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.132426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:53059 (local address 127.2.90.130:45109)
0620 22:52:15.132480 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.132482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.132551 (+ 69us) server_negotiation.cc:407] Connection header received
0620 22:52:15.132606 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.132607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.132618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.132669 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:15.132875 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.133610 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.135134 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.135696 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.135768 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:15.135776 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.135936 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.135965 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:52:15.136029 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.132438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:51287 (local address 127.2.90.131:42199)
0620 22:52:15.132481 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.132483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.132551 (+ 68us) server_negotiation.cc:407] Connection header received
0620 22:52:15.132606 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.132607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.132618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.132669 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:15.132875 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.133617 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.135134 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.135696 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.135768 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:15.135776 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.135925 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.135963 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:52:15.136055 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.132346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45109 (local address 127.2.90.129:53059)
0620 22:52:15.132470 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.132479 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.132542 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.132701 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.132702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.132704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:15.132808 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:15.132811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.133691 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.133692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:15.135051 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:15.135054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.135773 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.135774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:15.135857 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:15.135859 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.135910 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.135952 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":107,"thread_start_us":56,"threads_started":1}
I20260620 22:52:15.136062 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.132346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42199 (local address 127.2.90.129:51287)
0620 22:52:15.132468 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.132479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.132542 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.132701 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.132702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.132705 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:15.132808 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:15.132811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.133691 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.133692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:15.135051 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:15.135054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.135775 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.135776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:15.135853 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:15.135858 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.135910 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.135952 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":73,"thread_start_us":60,"threads_started":1}
I20260620 22:52:15.136451 6770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ededfcf199f400bbe29be82567eadaa" candidate_uuid: "b3000d836f8b4d6c9af444c901e08bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a753a8ccdcb5405cbad8a55b9742165c" is_pre_election: true
I20260620 22:52:15.136484 6903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ededfcf199f400bbe29be82567eadaa" candidate_uuid: "b3000d836f8b4d6c9af444c901e08bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" is_pre_election: true
I20260620 22:52:15.136579 6770 raft_consensus.cc:2454] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3000d836f8b4d6c9af444c901e08bb3 in term 0.
I20260620 22:52:15.136613 6903 raft_consensus.cc:2454] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3000d836f8b4d6c9af444c901e08bb3 in term 0.
I20260620 22:52:15.136804 6572 leader_election.cc:302] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a753a8ccdcb5405cbad8a55b9742165c, b3000d836f8b4d6c9af444c901e08bb3; no voters:
I20260620 22:52:15.136922 6970 raft_consensus.cc:2781] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:15.136976 6970 raft_consensus.cc:482] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:15.136996 6970 raft_consensus.cc:3037] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.137588 6970 raft_consensus.cc:504] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:15.137694 6970 leader_election.cc:288] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [CANDIDATE]: Term 1 election: Requested vote from peers a753a8ccdcb5405cbad8a55b9742165c (127.2.90.130:45109), 59dce15d7ffa4d0c842e00e109c5b35b (127.2.90.131:42199)
I20260620 22:52:15.137835 6770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ededfcf199f400bbe29be82567eadaa" candidate_uuid: "b3000d836f8b4d6c9af444c901e08bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a753a8ccdcb5405cbad8a55b9742165c"
I20260620 22:52:15.137856 6903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ededfcf199f400bbe29be82567eadaa" candidate_uuid: "b3000d836f8b4d6c9af444c901e08bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59dce15d7ffa4d0c842e00e109c5b35b"
I20260620 22:52:15.137908 6770 raft_consensus.cc:3037] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.137913 6903 raft_consensus.cc:3037] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.138556 6770 raft_consensus.cc:2454] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3000d836f8b4d6c9af444c901e08bb3 in term 1.
I20260620 22:52:15.138561 6903 raft_consensus.cc:2454] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3000d836f8b4d6c9af444c901e08bb3 in term 1.
I20260620 22:52:15.138731 6573 leader_election.cc:302] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59dce15d7ffa4d0c842e00e109c5b35b, b3000d836f8b4d6c9af444c901e08bb3; no voters:
I20260620 22:52:15.138808 6970 raft_consensus.cc:2781] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:15.138939 6970 raft_consensus.cc:686] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [term 1 LEADER]: Becoming Leader. State: Replica: b3000d836f8b4d6c9af444c901e08bb3, State: Running, Role: LEADER
I20260620 22:52:15.139017 6970 consensus_queue.cc:237] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } }
I20260620 22:52:15.139789 6492 catalog_manager.cc:5495] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3000d836f8b4d6c9af444c901e08bb3 (127.2.90.129). New cstate: current_term: 1 leader_uuid: "b3000d836f8b4d6c9af444c901e08bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3000d836f8b4d6c9af444c901e08bb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 } health_report { overall_health: UNKNOWN } } }
I20260620 22:52:15.193207 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.191081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.2.90.129:36533)
0620 22:52:15.191136 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.191138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.191146 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:15.191180 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.191180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.191216 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.191276 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:15.191404 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.191874 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.192606 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:15.192799 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:15.192842 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:15.192955 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:15.193051 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:15.193106 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.193119 (+ 13us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.193134 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:15.194855 6770 raft_consensus.cc:1261] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 1 FOLLOWER]: Refusing update from remote peer b3000d836f8b4d6c9af444c901e08bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:15.194854 6903 raft_consensus.cc:1261] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 1 FOLLOWER]: Refusing update from remote peer b3000d836f8b4d6c9af444c901e08bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:15.195182 6974 consensus_queue.cc:1035] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a753a8ccdcb5405cbad8a55b9742165c" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:15.195304 6970 consensus_queue.cc:1035] T 4ededfcf199f400bbe29be82567eadaa P b3000d836f8b4d6c9af444c901e08bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59dce15d7ffa4d0c842e00e109c5b35b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:15.197767 6978 mvcc.cc:205] Tried to move back new op lower bound from 7299055350555734016 to 7299055350330032128. Current Snapshot: MvccSnapshot[applied={T|T < 7299055350555734016}]
I20260620 22:52:15.199872 6493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:15.199970 6493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:15.200640 6493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44329ca1df014aeba8be9bd3446570af: Sending DeleteTablet for 3 replicas of tablet 4ededfcf199f400bbe29be82567eadaa
I20260620 22:52:15.200990 6750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ededfcf199f400bbe29be82567eadaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:15 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:32874
I20260620 22:52:15.200999 6617 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ededfcf199f400bbe29be82567eadaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:15 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:56674
I20260620 22:52:15.201213 6883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ededfcf199f400bbe29be82567eadaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:15 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:55088
I20260620 22:52:15.201370 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6555
I20260620 22:52:15.201408 6987 tablet_replica.cc:330] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: stopping tablet replica
I20260620 22:52:15.201453 6988 tablet_replica.cc:330] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: stopping tablet replica
I20260620 22:52:15.201494 6987 raft_consensus.cc:2229] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:15.201539 6988 raft_consensus.cc:2229] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:15.201586 6987 raft_consensus.cc:2258] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:15.201622 6988 raft_consensus.cc:2258] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:15.202577 6988 ts_tablet_manager.cc:1905] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:15.202973 6987 ts_tablet_manager.cc:1905] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:15.204108 6988 ts_tablet_manager.cc:1918] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:15.204169 6988 log.cc:1196] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/4ededfcf199f400bbe29be82567eadaa
I20260620 22:52:15.204387 6988 ts_tablet_manager.cc:1939] T 4ededfcf199f400bbe29be82567eadaa P 59dce15d7ffa4d0c842e00e109c5b35b: Deleting consensus metadata
I20260620 22:52:15.204383 6987 ts_tablet_manager.cc:1918] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:15.204450 6987 log.cc:1196] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/4ededfcf199f400bbe29be82567eadaa
I20260620 22:52:15.204669 6987 ts_tablet_manager.cc:1939] T 4ededfcf199f400bbe29be82567eadaa P a753a8ccdcb5405cbad8a55b9742165c: Deleting consensus metadata
I20260620 22:52:15.204814 6480 catalog_manager.cc:4841] TS 59dce15d7ffa4d0c842e00e109c5b35b (127.2.90.131:42199): tablet 4ededfcf199f400bbe29be82567eadaa (table test-table [id=d4ac1676d51d4ae9974cae7585743078]) successfully deleted
I20260620 22:52:15.205024 6479 catalog_manager.cc:4841] TS a753a8ccdcb5405cbad8a55b9742165c (127.2.90.130:45109): tablet 4ededfcf199f400bbe29be82567eadaa (table test-table [id=d4ac1676d51d4ae9974cae7585743078]) successfully deleted
W20260620 22:52:15.207218 6479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:36533 (error 108)
I20260620 22:52:15.207476 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6688
I20260620 22:52:15.207628 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.207507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36533 (local address 127.0.0.1:56696)
0620 22:52:15.207561 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.207600 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:36533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 22:52:15.208165 6479 catalog_manager.cc:4584] TS b3000d836f8b4d6c9af444c901e08bb3 (127.2.90.129:36533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ededfcf199f400bbe29be82567eadaa: Network error: Client connection negotiation failed: client connection to 127.2.90.129:36533: connect: Connection refused (error 111)
I20260620 22:52:15.211611 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6821
I20260620 22:52:15.215412 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6462
2026-06-20T22:52:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1992 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T22:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:15Z Disabled control of system clock
I20260620 22:52:15.239768 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37407
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:37407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:52:15.317287 6993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:15.317471 6993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:15.317492 6993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:15.318540 6993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:15.318570 6993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:15.318580 6993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:15.318591 6993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:15.318602 6993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:15.319836 6993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:37407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:15.320010 6993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:15.320223 6993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:15.324579 6999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.324571 6998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:15.324631 6993 server_base.cc:1034] running on GCE node
W20260620 22:52:15.324587 7001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:15.325006 6993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:15.325258 6993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:15.326385 6993 hybrid_clock.cc:648] HybridClock initialized: now 1781995935326372 us; error 30 us; skew 500 ppm
I20260620 22:52:15.327476 6993 webserver.cc:460] Webserver started at http://127.2.90.190:45803/ using document root <none> and password file <none>
I20260620 22:52:15.327680 6993 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:15.327723 6993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:15.327838 6993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:15.328874 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "348bc21bbbe64bc39ca194a5a3c099a5"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "2dc68cd14ff6365a1fe3119516b85ed5"
server_key_iv: "41601de8aecfa5e47d80188e13ad06c6"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.329185 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "348bc21bbbe64bc39ca194a5a3c099a5"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "2dc68cd14ff6365a1fe3119516b85ed5"
server_key_iv: "41601de8aecfa5e47d80188e13ad06c6"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.330377 6993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:15.331054 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.331202 6993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:15.331280 6993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "348bc21bbbe64bc39ca194a5a3c099a5"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "2dc68cd14ff6365a1fe3119516b85ed5"
server_key_iv: "41601de8aecfa5e47d80188e13ad06c6"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.331339 6993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:15.334501 6993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:15.334736 6993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:15.334849 6993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:15.338330 6993 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:37407
I20260620 22:52:15.338358 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:37407 every 8 connection(s)
I20260620 22:52:15.338647 6993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:15.339203 7060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:15.341826 7060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5: Bootstrap starting.
I20260620 22:52:15.342338 7060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:15.342595 7060 log.cc:824] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:15.343179 7060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5: No bootstrap required, opened a new log
I20260620 22:52:15.343225 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 6993
I20260620 22:52:15.343309 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:15.343463 2410 external_mini_cluster.cc:1428] Setting key 07eca6fb65dc1c7035c93bbf3c9274ff
I20260620 22:52:15.345184 7060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37407 } }
I20260620 22:52:15.345345 7060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:15.345378 7060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348bc21bbbe64bc39ca194a5a3c099a5, State: Initialized, Role: FOLLOWER
I20260620 22:52:15.345507 7060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37407 } }
I20260620 22:52:15.345587 7060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:15.345620 7060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:15.345657 7060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.346206 7060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37407 } }
I20260620 22:52:15.346304 7060 leader_election.cc:302] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 348bc21bbbe64bc39ca194a5a3c099a5; no voters:
I20260620 22:52:15.346439 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.343945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.2.90.190:37407)
0620 22:52:15.344133 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.344142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.344173 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:52:15.345149 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.345158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.345232 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.345380 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.346005 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.346008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.346009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.346013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.346057 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.346071 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.346144 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.346177 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.346231 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260620 22:52:15.346463 7060 leader_election.cc:288] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:15.346675 7060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:15.346715 7065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:15.346840 7065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [term 1 LEADER]: Becoming Leader. State: Replica: 348bc21bbbe64bc39ca194a5a3c099a5, State: Running, Role: LEADER
I20260620 22:52:15.346922 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37407 } }
I20260620 22:52:15.347234 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37407 } } }
I20260620 22:52:15.347330 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:15.347234 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 348bc21bbbe64bc39ca194a5a3c099a5. Latest consensus state: current_term: 1 leader_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348bc21bbbe64bc39ca194a5a3c099a5" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37407 } } }
I20260620 22:52:15.347406 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:15.347589 7074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:15.348023 7074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:15.349407 7074 catalog_manager.cc:1347] Generated new cluster ID: 8a6218727d704ebb92219b95a00dda4c
I20260620 22:52:15.349464 7074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:15.354600 7074 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:15.354732 7074 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:15.359339 7074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5: Generated new TSK 0
I20260620 22:52:15.359513 7074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:15.364333 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37407
--builtin_ntp_servers=127.2.90.148:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 22:52:15.444357 7084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:15.444561 7084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:15.444581 7084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:15.445740 7084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:15.445778 7084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:15.445803 7084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:15.447170 7084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:15.447377 7084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:15.447667 7084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:15.452310 7092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.452323 7089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.452323 7090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:15.452721 7084 server_base.cc:1034] running on GCE node
I20260620 22:52:15.452927 7084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:15.453155 7084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:15.454295 7084 hybrid_clock.cc:648] HybridClock initialized: now 1781995935454281 us; error 31 us; skew 500 ppm
I20260620 22:52:15.455422 7084 webserver.cc:460] Webserver started at http://127.2.90.129:33883/ using document root <none> and password file <none>
I20260620 22:52:15.455643 7084 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:15.455691 7084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:15.455816 7084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:15.456873 7084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "02a8e9b01f185c567fe3e8036287bbf8"
server_key_iv: "f6d961db20e5813d3c28618e4b461540"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.457226 7084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "02a8e9b01f185c567fe3e8036287bbf8"
server_key_iv: "f6d961db20e5813d3c28618e4b461540"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.458442 7084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:15.459157 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.459316 7084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:15.459400 7084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "02a8e9b01f185c567fe3e8036287bbf8"
server_key_iv: "f6d961db20e5813d3c28618e4b461540"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.459460 7084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:15.462579 7084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:15.462806 7084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:15.462929 7084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:15.463114 7084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:15.463383 7084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:15.463415 7084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.463440 7084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:15.463456 7084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.468701 7084 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:43343
I20260620 22:52:15.468748 7211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:43343 every 8 connection(s)
I20260620 22:52:15.469040 7084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:15.471338 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7084
I20260620 22:52:15.471438 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:15.471612 2410 external_mini_cluster.cc:1428] Setting key 2882c39a3532767c55c9c22948ad91d2
I20260620 22:52:15.471804 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.469656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:32993 (local address 127.2.90.190:37407)
0620 22:52:15.469699 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.469701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.469929 (+ 228us) server_negotiation.cc:407] Connection header received
0620 22:52:15.470374 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.470375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.470381 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.470428 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.471557 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.471558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.471558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.471563 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.471577 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.471581 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.471621 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.471724 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.471739 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:15.472184 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.469672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37407 (local address 127.2.90.129:32993)
0620 22:52:15.469832 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.469863 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.470296 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.470491 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.470495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.470502 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.471376 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.471377 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.471416 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.471465 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.471642 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.471647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.471650 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.471725 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.471787 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260620 22:52:15.472512 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37407
--builtin_ntp_servers=127.2.90.148:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 22:52:15.472590 7212 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37407
I20260620 22:52:15.472671 7212 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:15.472826 7212 heartbeater.cc:511] Master 127.2.90.190:37407 requested a full tablet report, sending...
I20260620 22:52:15.473284 7024 ts_manager.cc:194] Registered new tserver with Master: 6c901d776d7c4cd1ab8e0836fbfd8b61 (127.2.90.129:43343)
W20260620 22:52:15.549665 7215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:15.549866 7215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:15.549896 7215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:15.551017 7215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:15.551056 7215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:15.551088 7215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:15.552395 7215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:15.552634 7215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:15.552920 7215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:15.557343 7220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.557345 7223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.557404 7221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:15.557566 7215 server_base.cc:1034] running on GCE node
I20260620 22:52:15.557761 7215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:15.557981 7215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:15.559127 7215 hybrid_clock.cc:648] HybridClock initialized: now 1781995935559116 us; error 30 us; skew 500 ppm
I20260620 22:52:15.560246 7215 webserver.cc:460] Webserver started at http://127.2.90.130:37909/ using document root <none> and password file <none>
I20260620 22:52:15.560482 7215 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:15.560528 7215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:15.560650 7215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:15.561745 7215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "39cd72791d134ed7b14e8df89371a4fa"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "8b153dce4172babf196bb2a969ed7690"
server_key_iv: "3e1594ad0faf894041791aa1d5e80736"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.562067 7215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "39cd72791d134ed7b14e8df89371a4fa"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "8b153dce4172babf196bb2a969ed7690"
server_key_iv: "3e1594ad0faf894041791aa1d5e80736"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.563247 7215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:15.563869 7229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.564030 7215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:15.564106 7215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "39cd72791d134ed7b14e8df89371a4fa"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "8b153dce4172babf196bb2a969ed7690"
server_key_iv: "3e1594ad0faf894041791aa1d5e80736"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.564164 7215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:15.567471 7215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:15.567696 7215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:15.567808 7215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:15.567987 7215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:15.568266 7215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:15.568298 7215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.568323 7215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:15.568339 7215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.573618 7215 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:46433
I20260620 22:52:15.573668 7342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:46433 every 8 connection(s)
I20260620 22:52:15.573948 7215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:15.576543 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.574550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:43457 (local address 127.2.90.190:37407)
0620 22:52:15.574611 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.574612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.574830 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:52:15.575260 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.575261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.575266 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.575331 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.576330 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.576331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.576332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.576335 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.576350 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.576354 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.576398 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.576479 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.576491 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:52:15.576614 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7215
I20260620 22:52:15.576719 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:15.576784 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.574536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37407 (local address 127.2.90.130:43457)
0620 22:52:15.574715 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.574752 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.575185 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.575383 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.575386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.575395 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.576204 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.576204 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.576231 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.576258 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.576405 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.576408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.576417 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.576487 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.576550 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260620 22:52:15.576906 2410 external_mini_cluster.cc:1428] Setting key a13f17e46b5890953341988343c75cba
I20260620 22:52:15.577395 7343 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37407
I20260620 22:52:15.577486 7343 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:15.577631 7343 heartbeater.cc:511] Master 127.2.90.190:37407 requested a full tablet report, sending...
I20260620 22:52:15.577832 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37407
--builtin_ntp_servers=127.2.90.148:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 22:52:15.578920 7024 ts_manager.cc:194] Registered new tserver with Master: 39cd72791d134ed7b14e8df89371a4fa (127.2.90.130:46433)
W20260620 22:52:15.655619 7346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:15.655808 7346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:15.655828 7346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:15.657032 7346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:15.657071 7346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:15.657095 7346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:15.658370 7346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:15.658546 7346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:15.658764 7346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:15.663326 7351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.663321 7354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.663319 7352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:15.663396 7346 server_base.cc:1034] running on GCE node
I20260620 22:52:15.663735 7346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:15.663965 7346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:15.665097 7346 hybrid_clock.cc:648] HybridClock initialized: now 1781995935665087 us; error 24 us; skew 500 ppm
I20260620 22:52:15.666196 7346 webserver.cc:460] Webserver started at http://127.2.90.131:42061/ using document root <none> and password file <none>
I20260620 22:52:15.666497 7346 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:15.666553 7346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:15.666678 7346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:15.667696 7346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "97b8159c39e94efead3cb2a520d8ba42"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "23fad5a49f54869f2b188515dce3c3fe"
server_key_iv: "b85a1320183ede54d9d84b0d5bb719d9"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.668011 7346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "97b8159c39e94efead3cb2a520d8ba42"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "23fad5a49f54869f2b188515dce3c3fe"
server_key_iv: "b85a1320183ede54d9d84b0d5bb719d9"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.669225 7346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:15.669901 7360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.670053 7346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:15.670133 7346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "97b8159c39e94efead3cb2a520d8ba42"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "23fad5a49f54869f2b188515dce3c3fe"
server_key_iv: "b85a1320183ede54d9d84b0d5bb719d9"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.670198 7346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:15.673343 7346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:15.673578 7346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:15.673689 7346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:15.673877 7346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:15.674141 7346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:15.674171 7346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.674196 7346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:15.674214 7346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.679360 7346 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:35577
I20260620 22:52:15.679395 7473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:35577 every 8 connection(s)
I20260620 22:52:15.679678 7346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:15.682022 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7346
I20260620 22:52:15.682104 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:15.682276 2410 external_mini_cluster.cc:1428] Setting key 09d0ff8eb57eacb50132af3ff6c9e9d4
I20260620 22:52:15.682368 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.680293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37143 (local address 127.2.90.190:37407)
0620 22:52:15.680344 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.680346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.680565 (+ 219us) server_negotiation.cc:407] Connection header received
0620 22:52:15.681026 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.681027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.681032 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.681086 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.682142 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.682143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.682144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.682148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.682164 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.682169 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.682215 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.682288 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.682303 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:52:15.682531 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.680274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37407 (local address 127.2.90.131:37143)
0620 22:52:15.680453 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.680489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.680948 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.681135 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.681138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.681146 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.681998 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.681999 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.682024 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.682051 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.682214 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.682219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.682221 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.682289 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.682347 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260620 22:52:15.682857 7474 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37407
I20260620 22:52:15.682937 7474 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:15.683069 7474 heartbeater.cc:511] Master 127.2.90.190:37407 requested a full tablet report, sending...
I20260620 22:52:15.683374 7024 ts_manager.cc:194] Registered new tserver with Master: 97b8159c39e94efead3cb2a520d8ba42 (127.2.90.131:35577)
I20260620 22:52:15.693773 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:52:15.695839 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.694997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.2.90.190:37407)
0620 22:52:15.695050 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.695051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.695143 (+ 92us) server_negotiation.cc:407] Connection header received
0620 22:52:15.695183 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.695183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.695188 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.695231 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.695674 (+ 443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.695674 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.695674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.695677 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.695690 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.695694 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.695733 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.695762 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.695774 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:52:15.697178 7024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:15.698568 7024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:15.701798 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.700353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:46433 (local address 127.0.0.1:34806)
0620 22:52:15.700650 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.700659 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.700728 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.700897 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.700900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.700906 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.701408 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.701408 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.701432 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.701454 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.701648 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.701653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.701656 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.701716 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.701728 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":4,"spinlock_wait_cycles":384,"thread_start_us":85,"threads_started":1}
I20260620 22:52:15.701825 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.700438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.2.90.130:46433)
0620 22:52:15.700778 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.700781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.700790 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:52:15.700811 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.700814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.700823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.700884 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.701517 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.701524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.701525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.701529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.701563 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.701585 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.701644 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.701725 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.701740 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":82,"threads_started":1}
I20260620 22:52:15.702140 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.700352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:43343 (local address 127.0.0.1:37764)
0620 22:52:15.700617 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.700658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.700737 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.701261 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.701263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.701266 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.701777 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.701778 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.701782 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.701789 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.702029 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.702030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.702030 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.702070 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.702082 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":62,"threads_started":1}
I20260620 22:52:15.702252 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.700615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.2.90.131:35577)
0620 22:52:15.700804 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.700808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.700941 (+ 133us) server_negotiation.cc:407] Connection header received
0620 22:52:15.700963 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.700965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.700972 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.701027 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.701921 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.701928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.701928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.701932 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.701964 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.701986 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.702039 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.702168 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.702182 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260620 22:52:15.702327 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.700439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.2.90.129:43343)
0620 22:52:15.700668 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.700673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.700790 (+ 117us) server_negotiation.cc:407] Connection header received
0620 22:52:15.700818 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.700820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.700831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.700896 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.701859 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.701865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.701866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.701870 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.701908 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.701945 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.702004 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.702239 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.702254 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260620 22:52:15.702571 7277 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0e3f67ed0142748fd05276afc4fafd (DEFAULT_TABLE table=test-table [id=023a7847ff62481ea714d5bb32cdc05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:15.702821 7277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0e3f67ed0142748fd05276afc4fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:15.702809 7145 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0e3f67ed0142748fd05276afc4fafd (DEFAULT_TABLE table=test-table [id=023a7847ff62481ea714d5bb32cdc05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:15.703058 7145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0e3f67ed0142748fd05276afc4fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:15.704959 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.700541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35577 (local address 127.0.0.1:56228)
0620 22:52:15.700850 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.700860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.700897 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.701262 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.701263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.701265 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.701686 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.701687 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.701693 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.701705 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.702065 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.702066 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.702067 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.702107 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.702119 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":57,"threads_started":1}
I20260620 22:52:15.705215 7488 tablet_bootstrap.cc:492] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Bootstrap starting.
I20260620 22:52:15.705569 7489 tablet_bootstrap.cc:492] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: Bootstrap starting.
I20260620 22:52:15.705765 7408 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0e3f67ed0142748fd05276afc4fafd (DEFAULT_TABLE table=test-table [id=023a7847ff62481ea714d5bb32cdc05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:15.706032 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d0e3f67ed0142748fd05276afc4fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:15.706089 7488 tablet_bootstrap.cc:654] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:15.706121 7489 tablet_bootstrap.cc:654] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:15.706398 7488 log.cc:824] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:15.706398 7489 log.cc:824] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:15.707064 7488 tablet_bootstrap.cc:492] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: No bootstrap required, opened a new log
I20260620 22:52:15.707134 7488 ts_tablet_manager.cc:1397] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:15.707329 7489 tablet_bootstrap.cc:492] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: No bootstrap required, opened a new log
I20260620 22:52:15.707391 7489 ts_tablet_manager.cc:1397] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:15.708591 7492 tablet_bootstrap.cc:492] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Bootstrap starting.
I20260620 22:52:15.709127 7492 tablet_bootstrap.cc:654] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:15.709371 7492 log.cc:824] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:15.709764 7488 raft_consensus.cc:348] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.709764 7489 raft_consensus.cc:348] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.709931 7488 raft_consensus.cc:374] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:15.709931 7489 raft_consensus.cc:374] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:15.709945 7492 tablet_bootstrap.cc:492] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: No bootstrap required, opened a new log
I20260620 22:52:15.709966 7488 raft_consensus.cc:729] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cd72791d134ed7b14e8df89371a4fa, State: Initialized, Role: FOLLOWER
I20260620 22:52:15.709966 7489 raft_consensus.cc:729] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c901d776d7c4cd1ab8e0836fbfd8b61, State: Initialized, Role: FOLLOWER
I20260620 22:52:15.709996 7492 ts_tablet_manager.cc:1397] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:15.710093 7489 consensus_queue.cc:260] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.710093 7488 consensus_queue.cc:260] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.710323 7489 ts_tablet_manager.cc:1428] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:52:15.710323 7488 ts_tablet_manager.cc:1428] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:52:15.710431 7343 heartbeater.cc:503] Master 127.2.90.190:37407 was elected leader, sending a full tablet report...
I20260620 22:52:15.710438 7212 heartbeater.cc:503] Master 127.2.90.190:37407 was elected leader, sending a full tablet report...
I20260620 22:52:15.712096 7492 raft_consensus.cc:348] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.712210 7492 raft_consensus.cc:374] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:15.712251 7492 raft_consensus.cc:729] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b8159c39e94efead3cb2a520d8ba42, State: Initialized, Role: FOLLOWER
I20260620 22:52:15.712354 7492 consensus_queue.cc:260] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.712565 7492 ts_tablet_manager.cc:1428] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:52:15.712620 7474 heartbeater.cc:503] Master 127.2.90.190:37407 was elected leader, sending a full tablet report...
W20260620 22:52:15.719480 7213 tablet.cc:2369] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:52:15.824445 7344 tablet.cc:2369] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:15.828853 7496 raft_consensus.cc:482] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:15.828977 7496 raft_consensus.cc:504] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.829313 7496 leader_election.cc:288] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39cd72791d134ed7b14e8df89371a4fa (127.2.90.130:46433), 6c901d776d7c4cd1ab8e0836fbfd8b61 (127.2.90.129:43343)
I20260620 22:52:15.830439 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.829499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:46433 (local address 127.2.90.131:35753)
0620 22:52:15.829540 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.829551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.829588 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.829726 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.829727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.829730 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.830129 (+ 399us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.830130 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.830138 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.830151 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.830314 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.830315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.830316 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.830360 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.830375 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:52:15.830446 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.829557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:35753 (local address 127.2.90.130:46433)
0620 22:52:15.829607 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.829609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.829620 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:15.829647 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.829648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.829653 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.829711 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.830226 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.830228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.830228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.830232 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.830248 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.830253 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.830310 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.830370 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.830383 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:52:15.830610 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.829563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:40527 (local address 127.2.90.129:43343)
0620 22:52:15.829607 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.829632 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.829873 (+ 241us) server_negotiation.cc:407] Connection header received
0620 22:52:15.829904 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.829905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.829910 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.829965 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.830366 (+ 401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.830367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.830368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.830372 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.830388 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.830392 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.830452 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.830537 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.830552 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:15.830777 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.829506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:43343 (local address 127.2.90.131:40527)
0620 22:52:15.829806 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.829814 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:15.829860 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:15.829967 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:15.829968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:15.829970 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:15.830271 (+ 301us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:15.830272 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:15.830278 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:15.830288 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:15.830457 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:15.830457 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:15.830458 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:52:15.830688 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 22:52:15.830702 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":126,"thread_start_us":118,"threads_started":1}
I20260620 22:52:15.830726 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0e3f67ed0142748fd05276afc4fafd" candidate_uuid: "97b8159c39e94efead3cb2a520d8ba42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cd72791d134ed7b14e8df89371a4fa" is_pre_election: true
I20260620 22:52:15.830870 7297 raft_consensus.cc:2454] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97b8159c39e94efead3cb2a520d8ba42 in term 0.
I20260620 22:52:15.831094 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0e3f67ed0142748fd05276afc4fafd" candidate_uuid: "97b8159c39e94efead3cb2a520d8ba42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" is_pre_election: true
I20260620 22:52:15.831116 7364 leader_election.cc:302] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39cd72791d134ed7b14e8df89371a4fa, 97b8159c39e94efead3cb2a520d8ba42; no voters:
I20260620 22:52:15.831223 7166 raft_consensus.cc:2454] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97b8159c39e94efead3cb2a520d8ba42 in term 0.
I20260620 22:52:15.831277 7496 raft_consensus.cc:2781] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:15.831336 7496 raft_consensus.cc:482] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:15.831357 7496 raft_consensus.cc:3037] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.832033 7496 raft_consensus.cc:504] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.832146 7496 leader_election.cc:288] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [CANDIDATE]: Term 1 election: Requested vote from peers 39cd72791d134ed7b14e8df89371a4fa (127.2.90.130:46433), 6c901d776d7c4cd1ab8e0836fbfd8b61 (127.2.90.129:43343)
I20260620 22:52:15.832283 7297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0e3f67ed0142748fd05276afc4fafd" candidate_uuid: "97b8159c39e94efead3cb2a520d8ba42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cd72791d134ed7b14e8df89371a4fa"
I20260620 22:52:15.832296 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0e3f67ed0142748fd05276afc4fafd" candidate_uuid: "97b8159c39e94efead3cb2a520d8ba42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61"
I20260620 22:52:15.832353 7166 raft_consensus.cc:3037] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.832356 7297 raft_consensus.cc:3037] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:15.833086 7166 raft_consensus.cc:2454] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97b8159c39e94efead3cb2a520d8ba42 in term 1.
I20260620 22:52:15.833122 7297 raft_consensus.cc:2454] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97b8159c39e94efead3cb2a520d8ba42 in term 1.
I20260620 22:52:15.833243 7363 leader_election.cc:302] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c901d776d7c4cd1ab8e0836fbfd8b61, 97b8159c39e94efead3cb2a520d8ba42; no voters:
I20260620 22:52:15.833379 7496 raft_consensus.cc:2781] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:15.833524 7496 raft_consensus.cc:686] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 1 LEADER]: Becoming Leader. State: Replica: 97b8159c39e94efead3cb2a520d8ba42, State: Running, Role: LEADER
I20260620 22:52:15.833609 7496 consensus_queue.cc:237] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } }
I20260620 22:52:15.834406 7023 catalog_manager.cc:5495] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97b8159c39e94efead3cb2a520d8ba42 (127.2.90.131). New cstate: current_term: 1 leader_uuid: "97b8159c39e94efead3cb2a520d8ba42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97b8159c39e94efead3cb2a520d8ba42" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35577 } health_report { overall_health: HEALTHY } } }
I20260620 22:52:15.845203 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.844476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.2.90.131:35577)
0620 22:52:15.844512 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:15.844513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:15.844521 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:15.844573 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:15.844573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:15.844583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:15.844623 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:15.845049 (+ 426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:15.845050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:15.845050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:15.845053 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:15.845066 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:15.845069 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:15.845108 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:15.845135 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 22:52:15.845145 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:15.846717 7297 raft_consensus.cc:1261] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 1 FOLLOWER]: Refusing update from remote peer 97b8159c39e94efead3cb2a520d8ba42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:15.846719 7166 raft_consensus.cc:1261] T 6d0e3f67ed0142748fd05276afc4fafd P 6c901d776d7c4cd1ab8e0836fbfd8b61 [term 1 FOLLOWER]: Refusing update from remote peer 97b8159c39e94efead3cb2a520d8ba42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:15.847010 7496 consensus_queue.cc:1035] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c901d776d7c4cd1ab8e0836fbfd8b61" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:15.847133 7498 consensus_queue.cc:1035] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cd72791d134ed7b14e8df89371a4fa" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:15.849227 7501 mvcc.cc:205] Tried to move back new op lower bound from 7299055353225912320 to 7299055353175097344. Current Snapshot: MvccSnapshot[applied={T|T < 7299055353225912320}]
I20260620 22:52:15.849712 7502 mvcc.cc:205] Tried to move back new op lower bound from 7299055353225912320 to 7299055353175097344. Current Snapshot: MvccSnapshot[applied={T|T < 7299055353225912320}]
I20260620 22:52:15.849953 7503 mvcc.cc:205] Tried to move back new op lower bound from 7299055353225912320 to 7299055353175097344. Current Snapshot: MvccSnapshot[applied={T|T < 7299055353225912320}]
I20260620 22:52:15.851720 7023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:15.851794 7023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:15.852424 7023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 348bc21bbbe64bc39ca194a5a3c099a5: Sending DeleteTablet for 3 replicas of tablet 6d0e3f67ed0142748fd05276afc4fafd
I20260620 22:52:15.852757 7277 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0e3f67ed0142748fd05276afc4fafd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:15 UTC) from {username='slave'} at 127.0.0.1:34806
I20260620 22:52:15.852779 7145 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0e3f67ed0142748fd05276afc4fafd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:15 UTC) from {username='slave'} at 127.0.0.1:37764
I20260620 22:52:15.852978 7510 tablet_replica.cc:330] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: stopping tablet replica
I20260620 22:52:15.852970 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0e3f67ed0142748fd05276afc4fafd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:15 UTC) from {username='slave'} at 127.0.0.1:56228
I20260620 22:52:15.853070 7510 raft_consensus.cc:2229] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:15.853165 7510 raft_consensus.cc:2258] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:15.853179 7512 tablet_replica.cc:330] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: stopping tablet replica
I20260620 22:52:15.853286 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7084
I20260620 22:52:15.853415 7512 raft_consensus.cc:2229] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:15.853586 7512 raft_consensus.cc:2258] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:15.853605 7510 ts_tablet_manager.cc:1905] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:15.854123 7512 ts_tablet_manager.cc:1905] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:15.855427 7510 ts_tablet_manager.cc:1918] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:15.855499 7510 log.cc:1196] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/6d0e3f67ed0142748fd05276afc4fafd
I20260620 22:52:15.855736 7510 ts_tablet_manager.cc:1939] T 6d0e3f67ed0142748fd05276afc4fafd P 39cd72791d134ed7b14e8df89371a4fa: Deleting consensus metadata
I20260620 22:52:15.855855 7512 ts_tablet_manager.cc:1918] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:15.855958 7512 log.cc:1196] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/6d0e3f67ed0142748fd05276afc4fafd
I20260620 22:52:15.856123 7011 catalog_manager.cc:4841] TS 39cd72791d134ed7b14e8df89371a4fa (127.2.90.130:46433): tablet 6d0e3f67ed0142748fd05276afc4fafd (table test-table [id=023a7847ff62481ea714d5bb32cdc05d]) successfully deleted
I20260620 22:52:15.856307 7512 ts_tablet_manager.cc:1939] T 6d0e3f67ed0142748fd05276afc4fafd P 97b8159c39e94efead3cb2a520d8ba42: Deleting consensus metadata
I20260620 22:52:15.856622 7011 catalog_manager.cc:4841] TS 97b8159c39e94efead3cb2a520d8ba42 (127.2.90.131:35577): tablet 6d0e3f67ed0142748fd05276afc4fafd (table test-table [id=023a7847ff62481ea714d5bb32cdc05d]) successfully deleted
W20260620 22:52:15.860626 7010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:43343 (error 108)
I20260620 22:52:15.860869 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7215
I20260620 22:52:15.861034 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:15.860931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:43343 (local address 127.0.0.1:37766)
0620 22:52:15.860975 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:15.861007 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:52:15.861609 7010 catalog_manager.cc:4584] TS 6c901d776d7c4cd1ab8e0836fbfd8b61 (127.2.90.129:43343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d0e3f67ed0142748fd05276afc4fafd: Network error: Client connection negotiation failed: client connection to 127.2.90.129:43343: connect: Connection refused (error 111)
I20260620 22:52:15.864953 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7346
I20260620 22:52:15.868943 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 6993
2026-06-20T22:52:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (653 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 22:52:15.886994 2410 mini_oidc.cc:152] Starting JWKS server
I20260620 22:52:15.887995 2410 webserver.cc:460] Webserver started at https://127.0.0.1:38965/ using document root <none> and password file <none>
I20260620 22:52:15.888075 2410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:52:15.888234 2410 webserver.cc:460] Webserver started at http://127.0.0.1:39107/ using document root <none> and password file <none>
2026-06-20T22:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:15Z Disabled control of system clock
I20260620 22:52:15.894918 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39591
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:39591
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38965/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/testchainca.cert with env {}
W20260620 22:52:15.970505 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:15.970692 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:15.970712 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:15.971824 7519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:15.971855 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:15.971866 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:15.971877 7519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:15.971900 7519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:52:15.971931 7519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:52:15.971942 7519 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38965/jwks.json
I20260620 22:52:15.973417 7519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:39591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38965/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:15.973601 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:15.973755 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:52:15.978140 7519 server_base.cc:1034] running on GCE node
W20260620 22:52:15.978111 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.978108 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:15.978116 7524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:15.978523 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:15.978780 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:15.979913 7519 hybrid_clock.cc:648] HybridClock initialized: now 1781995935979891 us; error 36 us; skew 500 ppm
I20260620 22:52:15.985342 7519 webserver.cc:460] Webserver started at http://127.2.90.190:40427/ using document root <none> and password file <none>
I20260620 22:52:15.985581 7519 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:15.985634 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:15.985760 7519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:15.986784 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "d2b63b96e2504d298babd2602dc27776"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "eb30cff062c2c28ef6473f0523e41445"
server_key_iv: "2aa56ff1ffd4ed26f9d586e777dd7e80"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.987104 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "d2b63b96e2504d298babd2602dc27776"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "eb30cff062c2c28ef6473f0523e41445"
server_key_iv: "2aa56ff1ffd4ed26f9d586e777dd7e80"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.988307 7519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:52:15.988993 7533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.989145 7519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:15.989236 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "d2b63b96e2504d298babd2602dc27776"
format_stamp: "Formatted at 2026-06-20 22:52:15 on dist-test-slave-4mtw"
server_key: "eb30cff062c2c28ef6473f0523e41445"
server_key_iv: "2aa56ff1ffd4ed26f9d586e777dd7e80"
server_key_version: "encryptionkey@0"
I20260620 22:52:15.989300 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:16.002969 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:16.003261 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:16.003387 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:16.006937 7519 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:39591
I20260620 22:52:16.006974 7585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:39591 every 8 connection(s)
I20260620 22:52:16.007288 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:16.008190 7586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:16.008365 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7519
I20260620 22:52:16.008460 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:16.008651 2410 external_mini_cluster.cc:1428] Setting key c11ae5da48e8e8a4dc6d152f09ce3e6f
I20260620 22:52:16.010754 7586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776: Bootstrap starting.
I20260620 22:52:16.011327 7586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:16.011611 7586 log.cc:824] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:16.012986 7586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776: No bootstrap required, opened a new log
I20260620 22:52:16.013769 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.009168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.2.90.190:39591)
0620 22:52:16.009363 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.009371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.009394 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:52:16.009787 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.009792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.009851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.009973 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:16.010177 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.011044 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.012030 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.012437 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.012544 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.013157 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:16.013160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:16.013160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:16.013316 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:16.013348 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:16.013358 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:16.013408 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:16.013509 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 22:52:16.013568 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 22:52:16.014592 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:52:16.014930 7586 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b63b96e2504d298babd2602dc27776" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39591 } }
I20260620 22:52:16.015093 7586 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:16.015131 7586 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b63b96e2504d298babd2602dc27776, State: Initialized, Role: FOLLOWER
I20260620 22:52:16.015244 7586 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b63b96e2504d298babd2602dc27776" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39591 } }
I20260620 22:52:16.015318 7586 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:16.015349 7586 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:16.015396 7586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:16.015904 7586 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b63b96e2504d298babd2602dc27776" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39591 } }
I20260620 22:52:16.016003 7586 leader_election.cc:302] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b63b96e2504d298babd2602dc27776; no voters:
I20260620 22:52:16.016150 7586 leader_election.cc:288] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:16.016188 7591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:16.016317 7591 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [term 1 LEADER]: Becoming Leader. State: Replica: d2b63b96e2504d298babd2602dc27776, State: Running, Role: LEADER
I20260620 22:52:16.016386 7586 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:16.016400 7591 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b63b96e2504d298babd2602dc27776" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39591 } }
I20260620 22:52:16.016772 7592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b63b96e2504d298babd2602dc27776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b63b96e2504d298babd2602dc27776" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39591 } } }
I20260620 22:52:16.016782 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b63b96e2504d298babd2602dc27776. Latest consensus state: current_term: 1 leader_uuid: "d2b63b96e2504d298babd2602dc27776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b63b96e2504d298babd2602dc27776" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39591 } } }
I20260620 22:52:16.016870 7592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.016873 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.017047 7599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:16.017500 7599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:16.018958 7599 catalog_manager.cc:1347] Generated new cluster ID: fcd66c6294ac44d89fafabd53bd83e77
I20260620 22:52:16.019001 7599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:16.028173 7599 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:16.028616 7599 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:16.030931 7599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2b63b96e2504d298babd2602dc27776: Generated new TSK 0
I20260620 22:52:16.031073 7599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 22:52:16.051614 7616 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 22:52:16.051723 7616 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 22:52:16.051873 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.036772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.2.90.190:39591)
0620 22:52:16.036824 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.036826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.036933 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:52:16.037001 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.037002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.037013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.037079 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:52:16.037197 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.037952 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.038666 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.038969 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.039033 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.039179 (+ 146us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:52:16.051675 (+ 12496us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 22:52:16.051806 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 22:52:16.051928 7589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57966: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 22:52:16.052323 7615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:16.036723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39591 (local address 127.0.0.1:57966)
0620 22:52:16.036872 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:16.036881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:16.036935 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:16.037092 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:16.037092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:16.037094 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:52:16.037143 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.037144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.038030 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.038030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.038600 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.038601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.039032 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.039032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.039074 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.039089 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:52:16.051885 (+ 12796us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 22:52:16.052064 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.90.190:39591: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":81,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.90.190:39591: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 22:52:16.052747 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7519
2026-06-20T22:52:16Z chronyd exiting
I20260620 22:52:16.087592 2410 test_util.cc:175] -----------------------------------------------
I20260620 22:52:16.087656 2410 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781995912840861-2410-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (201 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 22:52:16.088305 2410 mini_oidc.cc:152] Starting JWKS server
I20260620 22:52:16.089025 2410 webserver.cc:460] Webserver started at https://127.0.0.1:37059/ using document root <none> and password file <none>
I20260620 22:52:16.089067 2410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:52:16.089200 2410 webserver.cc:460] Webserver started at http://127.0.0.1:42205/ using document root <none> and password file <none>
2026-06-20T22:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:16Z Disabled control of system clock
I20260620 22:52:16.096143 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37775
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:37775
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/testchainca.cert with env {}
W20260620 22:52:16.173739 7622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:16.173940 7622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:16.173960 7622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:16.175093 7622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:16.175129 7622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:16.175141 7622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:16.175153 7622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:16.175164 7622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:52:16.175175 7622 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:52:16.175184 7622 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37059/jwks.json
I20260620 22:52:16.176618 7622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:37775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:16.176824 7622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:16.176981 7622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:16.181447 7630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:16.181492 7628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:16.181463 7622 server_base.cc:1034] running on GCE node
W20260620 22:52:16.181440 7627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:16.181852 7622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:16.182139 7622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:16.183274 7622 hybrid_clock.cc:648] HybridClock initialized: now 1781995936183262 us; error 30 us; skew 500 ppm
I20260620 22:52:16.184394 7622 webserver.cc:460] Webserver started at http://127.2.90.190:35377/ using document root <none> and password file <none>
I20260620 22:52:16.184603 7622 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:16.184648 7622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:16.184792 7622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:16.185813 7622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "857dbb2bc93842b08f90a58b2c2b8c8a"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "813c7a5e51816cd77a8ef4092d74241f"
server_key_iv: "54a27339aeb28734948f2141bfc54b2d"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.186141 7622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "857dbb2bc93842b08f90a58b2c2b8c8a"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "813c7a5e51816cd77a8ef4092d74241f"
server_key_iv: "54a27339aeb28734948f2141bfc54b2d"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.187315 7622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:16.187947 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:16.188088 7622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:16.188176 7622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "857dbb2bc93842b08f90a58b2c2b8c8a"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "813c7a5e51816cd77a8ef4092d74241f"
server_key_iv: "54a27339aeb28734948f2141bfc54b2d"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.188238 7622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:16.215554 7622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:16.215873 7622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:16.216015 7622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:16.219591 7622 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:37775
I20260620 22:52:16.219645 7688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:37775 every 8 connection(s)
I20260620 22:52:16.219961 7622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:16.220204 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7622
I20260620 22:52:16.220292 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:16.220480 2410 external_mini_cluster.cc:1428] Setting key ab1650747bab46fd50a4de23075e0e35
I20260620 22:52:16.221055 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:16.223507 7689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a: Bootstrap starting.
I20260620 22:52:16.224051 7689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:16.224293 7689 log.cc:824] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:16.225014 7689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a: No bootstrap required, opened a new log
I20260620 22:52:16.225703 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.220994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.2.90.190:37775)
0620 22:52:16.221247 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.221255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.221280 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:52:16.221693 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.221699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.221758 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.221884 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:16.222077 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.222943 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.223764 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.224205 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.224313 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.225030 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:16.225035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:16.225035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:16.225224 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:16.225265 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:16.225280 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:16.225353 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:16.225407 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:52:16.225494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260620 22:52:16.226673 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:52:16.227115 7689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37775 } }
I20260620 22:52:16.227247 7689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:16.227280 7689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857dbb2bc93842b08f90a58b2c2b8c8a, State: Initialized, Role: FOLLOWER
I20260620 22:52:16.227396 7689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37775 } }
I20260620 22:52:16.227475 7689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:16.227509 7689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:16.227550 7689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:16.228067 7689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37775 } }
I20260620 22:52:16.228157 7689 leader_election.cc:302] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857dbb2bc93842b08f90a58b2c2b8c8a; no voters:
I20260620 22:52:16.228307 7689 leader_election.cc:288] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:16.228350 7694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:16.228487 7694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [term 1 LEADER]: Becoming Leader. State: Replica: 857dbb2bc93842b08f90a58b2c2b8c8a, State: Running, Role: LEADER
I20260620 22:52:16.228523 7689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:16.228578 7694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37775 } }
I20260620 22:52:16.228921 7695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37775 } } }
I20260620 22:52:16.229015 7695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.228935 7696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857dbb2bc93842b08f90a58b2c2b8c8a. Latest consensus state: current_term: 1 leader_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857dbb2bc93842b08f90a58b2c2b8c8a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37775 } } }
I20260620 22:52:16.229161 7696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.229756 7701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:16.230142 7701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:16.231419 7701 catalog_manager.cc:1347] Generated new cluster ID: 77af351dc07c4005ad5ef4eacdabd9ec
I20260620 22:52:16.231469 7701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:16.249145 7701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:16.249655 7701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:16.254089 7701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 857dbb2bc93842b08f90a58b2c2b8c8a: Generated new TSK 0
I20260620 22:52:16.254244 7701 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 22:52:16.273836 7719 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 22:52:16.273903 7719 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 22:52:16.274240 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.260172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.2.90.190:37775)
0620 22:52:16.260216 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.260218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.260294 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:52:16.260343 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.260344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.260356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.260427 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:52:16.260527 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.261321 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.262028 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.262353 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.262426 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.262554 (+ 128us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:52:16.274025 (+ 11471us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 22:52:16.274161 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 22:52:16.274298 7692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 22:52:16.274296 7718 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:16.260095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37775 (local address 127.0.0.1:33190)
0620 22:52:16.260223 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:16.260233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:16.260286 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:16.260427 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:16.260427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:16.260429 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:52:16.260475 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.260476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.261386 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.261386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.261948 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.261950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.262423 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.262423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.262458 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.262462 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:52:16.274139 (+ 11677us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 22:52:16.274225 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.90.190:37775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260620 22:52:16.274535 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7622
2026-06-20T22:52:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 22:52:16.310935 2410 mini_oidc.cc:152] Starting JWKS server
I20260620 22:52:16.311571 2410 webserver.cc:460] Webserver started at https://127.0.0.1:41977/ using document root <none> and password file <none>
I20260620 22:52:16.311614 2410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 22:52:16.311741 2410 webserver.cc:460] Webserver started at http://127.0.0.1:40093/ using document root <none> and password file <none>
2026-06-20T22:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:16Z Disabled control of system clock
I20260620 22:52:16.318327 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:33157
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:33157
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 22:52:16.394156 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:16.394346 7725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:16.394366 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:16.395484 7725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:16.395521 7725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:16.395532 7725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:16.395555 7725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:16.395570 7725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:52:16.395593 7725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:52:16.395613 7725 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41977/jwks.json
I20260620 22:52:16.396955 7725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:33157
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:16.397169 7725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:16.397431 7725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:52:16.401814 7725 server_base.cc:1034] running on GCE node
W20260620 22:52:16.401721 7731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:16.401739 7733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:16.401733 7730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:16.402132 7725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:16.402398 7725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:16.403535 7725 hybrid_clock.cc:648] HybridClock initialized: now 1781995936403519 us; error 33 us; skew 500 ppm
I20260620 22:52:16.404593 7725 webserver.cc:460] Webserver started at http://127.2.90.190:40047/ using document root <none> and password file <none>
I20260620 22:52:16.404819 7725 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:16.404863 7725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:16.404951 7725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:16.405900 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "2cbd7008ae064816ba57a940f94b9aae"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "83ed3328efe37aa96106146703ac71aa"
server_key_iv: "a7ec33c73b1b0463618bb2c1fa62182b"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.406163 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "2cbd7008ae064816ba57a940f94b9aae"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "83ed3328efe37aa96106146703ac71aa"
server_key_iv: "a7ec33c73b1b0463618bb2c1fa62182b"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.407318 7725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:16.407872 7739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:16.408023 7725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:16.408090 7725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "2cbd7008ae064816ba57a940f94b9aae"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "83ed3328efe37aa96106146703ac71aa"
server_key_iv: "a7ec33c73b1b0463618bb2c1fa62182b"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.408139 7725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:16.418864 7725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:16.419082 7725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:16.419171 7725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:16.422642 7725 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:33157
I20260620 22:52:16.422668 7791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:33157 every 8 connection(s)
I20260620 22:52:16.422950 7725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:16.423566 7792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:16.426159 7792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae: Bootstrap starting.
I20260620 22:52:16.426688 7792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:16.426909 7792 log.cc:824] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:16.427495 7792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae: No bootstrap required, opened a new log
I20260620 22:52:16.429025 7792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd7008ae064816ba57a940f94b9aae" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33157 } }
I20260620 22:52:16.429152 7792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:16.429172 7792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbd7008ae064816ba57a940f94b9aae, State: Initialized, Role: FOLLOWER
I20260620 22:52:16.429286 7792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd7008ae064816ba57a940f94b9aae" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33157 } }
I20260620 22:52:16.429359 7792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:16.429387 7792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:16.429415 7792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:16.429857 7792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd7008ae064816ba57a940f94b9aae" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33157 } }
I20260620 22:52:16.429935 7792 leader_election.cc:302] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cbd7008ae064816ba57a940f94b9aae; no voters:
I20260620 22:52:16.430056 7792 leader_election.cc:288] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:16.430115 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:16.430256 7795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [term 1 LEADER]: Becoming Leader. State: Replica: 2cbd7008ae064816ba57a940f94b9aae, State: Running, Role: LEADER
I20260620 22:52:16.430279 7792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:16.430346 7795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd7008ae064816ba57a940f94b9aae" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33157 } }
I20260620 22:52:16.430670 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbd7008ae064816ba57a940f94b9aae. Latest consensus state: current_term: 1 leader_uuid: "2cbd7008ae064816ba57a940f94b9aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd7008ae064816ba57a940f94b9aae" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33157 } } }
I20260620 22:52:16.430749 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.430864 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cbd7008ae064816ba57a940f94b9aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd7008ae064816ba57a940f94b9aae" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33157 } } }
I20260620 22:52:16.430924 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.431258 7804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:16.431536 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7725
I20260620 22:52:16.431627 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:16.431720 7804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:16.431825 2410 external_mini_cluster.cc:1428] Setting key a9c71902c5c950834b2c3e4d29865b80
I20260620 22:52:16.433426 7804 catalog_manager.cc:1347] Generated new cluster ID: 623ed16dea2847fe8ffefed62405dba4
I20260620 22:52:16.433478 7804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:16.437773 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.432379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.2.90.190:33157)
0620 22:52:16.432557 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.432565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.432595 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:52:16.432727 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.432733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.432807 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.432936 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:16.433261 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.434591 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.435681 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.436261 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.436371 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.437034 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:16.437038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:16.437038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:16.437224 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:16.437261 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:16.437273 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:16.437332 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:16.437421 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 22:52:16.437490 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260620 22:52:16.438679 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:52:16.448781 7804 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:16.449229 7804 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:16.458714 7804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cbd7008ae064816ba57a940f94b9aae: Generated new TSK 0
I20260620 22:52:16.458885 7804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:16.471369 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.467889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.2.90.190:33157)
0620 22:52:16.467938 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.467941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.468039 (+ 98us) server_negotiation.cc:407] Connection header received
0620 22:52:16.468112 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.468113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.468123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.468179 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:52:16.468298 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.468836 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.469943 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.470126 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.470167 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.470331 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:52:16.471179 (+ 848us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 22:52:16.471303 (+ 124us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 22:52:16.471437 7814 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 22:52:16.471819 7821 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:16.467817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:33157 (local address 127.0.0.1:45450)
0620 22:52:16.467970 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:16.467981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:16.468047 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:16.468184 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:16.468185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:16.468187 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:52:16.468242 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.468244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.468913 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.468913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.469871 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.469875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.470192 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.470193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.470251 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.470256 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:52:16.471288 (+ 1032us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 22:52:16.471392 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.90.190:33157: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41977/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260620 22:52:16.472026 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7725
E20260620 22:52:16.481953 7822 webserver.cc:533] Webserver: sslize failed:
E20260620 22:52:16.482016 7822 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T22:52:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (187 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T22:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:16Z Disabled control of system clock
I20260620 22:52:16.504558 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:33147
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:33147
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/testchainca.cert with env {}
W20260620 22:52:16.580724 7826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:16.580919 7826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:16.580940 7826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:16.582039 7826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:16.582072 7826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:16.582083 7826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:16.582094 7826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:16.582104 7826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 22:52:16.582115 7826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 22:52:16.582127 7826 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 22:52:16.583521 7826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:33147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:16.583698 7826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:16.583854 7826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:16.588173 7834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:16.588217 7832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:16.588181 7831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:16.588194 7826 server_base.cc:1034] running on GCE node
I20260620 22:52:16.588551 7826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:16.588858 7826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:16.589995 7826 hybrid_clock.cc:648] HybridClock initialized: now 1781995936589982 us; error 30 us; skew 500 ppm
I20260620 22:52:16.591073 7826 webserver.cc:460] Webserver started at http://127.2.90.190:46575/ using document root <none> and password file <none>
I20260620 22:52:16.591290 7826 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:16.591336 7826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:16.591465 7826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:16.592433 7826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "7ab1ddb31891405ba7a8bec27a24d7a7"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "dbe65e214454dd84b8de677f0ef74d31"
server_key_iv: "d47378d9e6f0eda5143116032f81fb17"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.592772 7826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "7ab1ddb31891405ba7a8bec27a24d7a7"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "dbe65e214454dd84b8de677f0ef74d31"
server_key_iv: "d47378d9e6f0eda5143116032f81fb17"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.593891 7826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:16.594514 7840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:16.594658 7826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:16.594738 7826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "7ab1ddb31891405ba7a8bec27a24d7a7"
format_stamp: "Formatted at 2026-06-20 22:52:16 on dist-test-slave-4mtw"
server_key: "dbe65e214454dd84b8de677f0ef74d31"
server_key_iv: "d47378d9e6f0eda5143116032f81fb17"
server_key_version: "encryptionkey@0"
I20260620 22:52:16.594803 7826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:16.610211 7826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:16.610502 7826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:16.610630 7826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:16.614171 7826 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:33147
I20260620 22:52:16.614229 7892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:33147 every 8 connection(s)
I20260620 22:52:16.614521 7826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:16.615376 7893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:16.617514 7893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7: Bootstrap starting.
I20260620 22:52:16.618046 7893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:16.618265 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7826
I20260620 22:52:16.618290 7893 log.cc:824] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:16.618357 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:16.618535 2410 external_mini_cluster.cc:1428] Setting key f1cc740b6e7ef7ae92f44d5524dd671b
I20260620 22:52:16.618993 7893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7: No bootstrap required, opened a new log
I20260620 22:52:16.621109 7893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33147 } }
I20260620 22:52:16.621237 7893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:16.621268 7893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab1ddb31891405ba7a8bec27a24d7a7, State: Initialized, Role: FOLLOWER
I20260620 22:52:16.621381 7893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33147 } }
I20260620 22:52:16.621455 7893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:16.621487 7893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:16.621533 7893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:16.622028 7893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33147 } }
I20260620 22:52:16.622115 7893 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab1ddb31891405ba7a8bec27a24d7a7; no voters:
I20260620 22:52:16.622254 7893 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:16.622328 7898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:16.622473 7898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 7ab1ddb31891405ba7a8bec27a24d7a7, State: Running, Role: LEADER
I20260620 22:52:16.622498 7893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:16.622642 7898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33147 } }
I20260620 22:52:16.623005 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33147 } } }
I20260620 22:52:16.623090 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.623219 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab1ddb31891405ba7a8bec27a24d7a7. Latest consensus state: current_term: 1 leader_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab1ddb31891405ba7a8bec27a24d7a7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33147 } } }
I20260620 22:52:16.623270 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:16.624244 7907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:16.624329 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.618976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.2.90.190:33147)
0620 22:52:16.619172 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.619181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.619210 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:52:16.620060 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.620066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.620123 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.620242 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:16.620443 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.621336 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.622340 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.622926 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.623351 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.623675 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:16.623678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:16.623679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:16.623872 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:16.623905 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:16.623916 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:16.623965 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:16.624053 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:52:16.624134 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260620 22:52:16.624665 7907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:16.625625 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:52:16.626116 7907 catalog_manager.cc:1347] Generated new cluster ID: 7d583daf4b014acd94096c085539615c
I20260620 22:52:16.626163 7907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:16.635977 7907 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:16.636626 7907 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:16.643877 7907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ab1ddb31891405ba7a8bec27a24d7a7: Generated new TSK 0
I20260620 22:52:16.644032 7907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:17.075394 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:16.655518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.2.90.190:33147)
0620 22:52:16.655558 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:16.655560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:16.655604 (+ 44us) server_negotiation.cc:407] Connection header received
0620 22:52:16.655685 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:16.655686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:16.655695 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:16.655749 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0620 22:52:16.655858 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.656338 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.657434 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.657627 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.657673 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.657845 (+ 172us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 22:52:17.075184 (+417339us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 22:52:17.075333 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36346: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:52:17.075472 7897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36346: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 22:52:17.075539 7922 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:16.655408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:33147 (local address 127.0.0.1:36346)
0620 22:52:16.655549 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:16.655559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:16.655623 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:16.655749 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:16.655750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:16.655751 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 22:52:16.655804 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.655806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.656403 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.656404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.657364 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:16.657367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:16.657692 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:16.657693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:16.657767 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:16.657773 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 22:52:17.075325 (+417552us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 22:52:17.075455 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.90.190:33147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260620 22:52:17.075863 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7826
2026-06-20T22:52:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (594 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:17 dist-test-slave-4mtw krb5kdc[7934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:17 dist-test-slave-4mtw krb5kdc[7934](info): set up 2 sockets
Jun 20 22:52:17 dist-test-slave-4mtw krb5kdc[7934](info): commencing operation
krb5kdc: starting...
W20260620 22:52:18.188473 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:18.297824 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35447
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:18.375624 7947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:18.375818 7947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:18.375847 7947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:18.376999 7947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:18.377038 7947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:18.377055 7947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:18.377071 7947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:18.377086 7947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:18.378495 7947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:18.378738 7947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:18.378945 7947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:18.383343 7952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:18.383358 7953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.383447 7947 server_base.cc:1034] running on GCE node
W20260620 22:52:18.383340 7955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.383796 7947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:18.384075 7947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:18.385206 7947 hybrid_clock.cc:648] HybridClock initialized: now 1781995938385195 us; error 30 us; skew 500 ppm
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.387444 7947 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:18.387841 7947 webserver.cc:460] Webserver started at http://127.2.90.190:46157/ using document root <none> and password file <none>
I20260620 22:52:18.388055 7947 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:18.388105 7947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:18.388212 7947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:18.389235 7947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "1b542ec1be484d369d55f9c5ac5e628a"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "f41e9a21f2b31a3571545b1f8a8173c3"
server_key_iv: "977fa641c1031462c8dc4458885df78d"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.389539 7947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "1b542ec1be484d369d55f9c5ac5e628a"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "f41e9a21f2b31a3571545b1f8a8173c3"
server_key_iv: "977fa641c1031462c8dc4458885df78d"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.390684 7947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:18.391315 7962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.391475 7947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:18.391553 7947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "1b542ec1be484d369d55f9c5ac5e628a"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "f41e9a21f2b31a3571545b1f8a8173c3"
server_key_iv: "977fa641c1031462c8dc4458885df78d"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.391613 7947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:18.408763 7947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:18.410823 7947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:18.411000 7947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:18.414669 7947 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35447
I20260620 22:52:18.414702 8014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35447 every 8 connection(s)
I20260620 22:52:18.415030 7947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:18.415805 8015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:18.417943 8015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a: Bootstrap starting.
I20260620 22:52:18.418437 8015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:18.418663 8015 log.cc:824] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:18.419229 8015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a: No bootstrap required, opened a new log
I20260620 22:52:18.420737 8015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b542ec1be484d369d55f9c5ac5e628a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35447 } }
I20260620 22:52:18.420868 8015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:18.420889 8015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b542ec1be484d369d55f9c5ac5e628a, State: Initialized, Role: FOLLOWER
I20260620 22:52:18.421001 8015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b542ec1be484d369d55f9c5ac5e628a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35447 } }
I20260620 22:52:18.421073 8015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:18.421105 8015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:18.421137 8015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:18.421586 8015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b542ec1be484d369d55f9c5ac5e628a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35447 } }
I20260620 22:52:18.421666 8015 leader_election.cc:302] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b542ec1be484d369d55f9c5ac5e628a; no voters:
I20260620 22:52:18.421784 8015 leader_election.cc:288] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:18.421834 8018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:18.421998 8015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:18.422202 8018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [term 1 LEADER]: Becoming Leader. State: Replica: 1b542ec1be484d369d55f9c5ac5e628a, State: Running, Role: LEADER
I20260620 22:52:18.422297 8018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b542ec1be484d369d55f9c5ac5e628a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35447 } }
I20260620 22:52:18.422772 8020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b542ec1be484d369d55f9c5ac5e628a. Latest consensus state: current_term: 1 leader_uuid: "1b542ec1be484d369d55f9c5ac5e628a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b542ec1be484d369d55f9c5ac5e628a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35447 } } }
I20260620 22:52:18.422812 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 7947
I20260620 22:52:18.422803 8019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b542ec1be484d369d55f9c5ac5e628a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b542ec1be484d369d55f9c5ac5e628a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35447 } } }
I20260620 22:52:18.422912 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:18.422894 8020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:18.422910 8019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:18.423105 2410 external_mini_cluster.cc:1428] Setting key de34b00bd899301f5b7e7135a0ab59e9
I20260620 22:52:18.423115 8027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:18.423583 8027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:18.425074 8027 catalog_manager.cc:1347] Generated new cluster ID: e28104c9ff174d7a9ac384bf10b720c1
I20260620 22:52:18.425125 8027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:18.436226 8027 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:18.436851 8027 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:18.442026 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.423553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.2.90.190:35447)
0620 22:52:18.432772 (+ 9219us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.432782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.432821 (+ 39us) server_negotiation.cc:407] Connection header received
0620 22:52:18.432940 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.432947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.432999 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.433139 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.433726 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.434652 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.435389 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.435834 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.435952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.438641 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.438646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.438646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.438658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.440472 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.440836 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.440839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.440839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.440953 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.441128 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.441129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.441129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.441407 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.441543 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.441640 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.441786 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9166,"thread_start_us":61,"threads_started":1}
I20260620 22:52:18.445708 8027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b542ec1be484d369d55f9c5ac5e628a: Generated new TSK 0
I20260620 22:52:18.445863 8027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:18.471133 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35447
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:18.548468 8041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:18.548662 8041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:18.548704 8041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:18.549883 8041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:18.549926 8041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:18.549952 8041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:18.551393 8041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:18.551594 8041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:18.551802 8041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:18.556315 8047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.556470 8041 server_base.cc:1034] running on GCE node
W20260620 22:52:18.556334 8049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:18.556348 8046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.556794 8041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:18.557024 8041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:18.558152 8041 hybrid_clock.cc:648] HybridClock initialized: now 1781995938558141 us; error 28 us; skew 500 ppm
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.560624 8041 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:18.561066 8041 webserver.cc:460] Webserver started at http://127.2.90.129:45183/ using document root <none> and password file <none>
I20260620 22:52:18.561239 8041 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:18.561276 8041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:18.561364 8041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:18.562311 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "d418e80483fd4faf88863dc47950b9a7"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "dacbd117943a0c79cf10f1d3c1454db0"
server_key_iv: "030a647903428d47353a9d86899ab92f"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.562593 8041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "d418e80483fd4faf88863dc47950b9a7"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "dacbd117943a0c79cf10f1d3c1454db0"
server_key_iv: "030a647903428d47353a9d86899ab92f"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.563714 8041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:18.564328 8056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.564488 8041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:18.564565 8041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "d418e80483fd4faf88863dc47950b9a7"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "dacbd117943a0c79cf10f1d3c1454db0"
server_key_iv: "030a647903428d47353a9d86899ab92f"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.564615 8041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:18.577757 8041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:18.579353 8041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:18.579455 8041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:18.579627 8041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:18.579869 8041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:18.579896 8041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.579926 8041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:18.579941 8041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.584968 8041 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:41449
I20260620 22:52:18.585035 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:41449 every 8 connection(s)
I20260620 22:52:18.585301 8041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:18.585621 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8041
I20260620 22:52:18.585716 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:18.585932 2410 external_mini_cluster.cc:1428] Setting key f0e1fb3dbe102653e53adbf9eb6f679a
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:52:18.594270 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.585978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:50563 (local address 127.2.90.190:35447)
0620 22:52:18.586021 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.586023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.586240 (+ 217us) server_negotiation.cc:407] Connection header received
0620 22:52:18.587252 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.587253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.587268 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.587316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.587899 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.588467 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.589495 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.589792 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.589864 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.591757 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.591759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.591759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.591766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.593154 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.593508 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.593509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.593509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.593567 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.593772 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.593772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.593773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.593863 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.593908 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.594140 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.594207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:52:18.594444 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.585960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35447 (local address 127.2.90.129:50563)
0620 22:52:18.586130 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:18.586175 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:18.587153 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:18.587385 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:18.587389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:18.587632 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:18.587811 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:18.587815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.588543 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.588543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:18.589387 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:18.589390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.589865 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.589866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:18.589963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.590420 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:18.590428 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:18.591665 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:18.593223 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:18.593229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:18.593230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:18.593437 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:18.593626 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:18.593627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:18.593627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:18.593724 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:18.593907 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:18.593910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:18.594018 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 22:52:18.594120 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 22:52:18.594249 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260620 22:52:18.594877 8170 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35447
I20260620 22:52:18.594971 8170 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:18.595127 8170 heartbeater.cc:511] Master 127.2.90.190:35447 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
I20260620 22:52:18.595660 7979 ts_manager.cc:194] Registered new tserver with Master: d418e80483fd4faf88863dc47950b9a7 (127.2.90.129:41449)
I20260620 22:52:18.596246 7979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:50563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:18.608776 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35447
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:18.686997 8175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:18.687184 8175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:18.687206 8175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:18.688318 8175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:18.688361 8175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:18.688390 8175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:18.689854 8175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:18.690070 8175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:18.690277 8175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:18.695271 8180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:18.695273 8181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.695361 8175 server_base.cc:1034] running on GCE node
W20260620 22:52:18.695273 8183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.695655 8175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:18.695891 8175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:18.697026 8175 hybrid_clock.cc:648] HybridClock initialized: now 1781995938697010 us; error 27 us; skew 500 ppm
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.699322 8175 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:18.699689 8175 webserver.cc:460] Webserver started at http://127.2.90.130:43223/ using document root <none> and password file <none>
I20260620 22:52:18.699851 8175 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:18.699887 8175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:18.699977 8175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:18.700958 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "03b82dee394c4328a1b215c55a8456a8"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "b1c3c52e40b9b74474b725f24da71fdf"
server_key_iv: "0669a99214ef1aa2c2355b37b8d029d1"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.701238 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "03b82dee394c4328a1b215c55a8456a8"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "b1c3c52e40b9b74474b725f24da71fdf"
server_key_iv: "0669a99214ef1aa2c2355b37b8d029d1"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.702339 8175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:18.702981 8190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.703143 8175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:18.703218 8175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "03b82dee394c4328a1b215c55a8456a8"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "b1c3c52e40b9b74474b725f24da71fdf"
server_key_iv: "0669a99214ef1aa2c2355b37b8d029d1"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.703267 8175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:18.724959 8175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:18.726631 8175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:18.726742 8175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:18.726931 8175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:18.727173 8175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:18.727202 8175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.727222 8175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:18.727238 8175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.732249 8175 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:39083
I20260620 22:52:18.732290 8303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:39083 every 8 connection(s)
I20260620 22:52:18.732581 8175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:18.732892 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8175
I20260620 22:52:18.732988 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:18.733163 2410 external_mini_cluster.cc:1428] Setting key 9be9ef046a939d6e5e9d0fd8678d35f5
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
I20260620 22:52:18.742561 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.733606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:60791 (local address 127.2.90.190:35447)
0620 22:52:18.733653 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.733655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.733766 (+ 111us) server_negotiation.cc:407] Connection header received
0620 22:52:18.734732 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.734732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.734744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.734815 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.735588 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.736087 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.737538 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.737741 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.737801 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.740030 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.740031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.740032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.740038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.741353 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.741717 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.741718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.741718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.741772 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.741979 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.741979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.741980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.742080 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.742133 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.742401 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.742474 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:52:18.742717 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.733303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35447 (local address 127.2.90.130:60791)
0620 22:52:18.733544 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:18.733672 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:18.734620 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:18.734879 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:18.734883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:18.735228 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:18.735501 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:18.735506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.736178 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.736179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:18.737441 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:18.737444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.737836 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.737838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:18.737983 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.738464 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:18.738474 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:18.739940 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:18.741431 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:18.741437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:18.741439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:18.741648 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:18.741828 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:18.741829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:18.741830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:18.741922 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:18.742135 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:18.742141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:18.742279 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:52:18.742364 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:52:18.742509 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
I20260620 22:52:18.743131 8304 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35447
I20260620 22:52:18.743207 8304 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:18.743376 8304 heartbeater.cc:511] Master 127.2.90.190:35447 requested a full tablet report, sending...
I20260620 22:52:18.743733 7979 ts_manager.cc:194] Registered new tserver with Master: 03b82dee394c4328a1b215c55a8456a8 (127.2.90.130:39083)
I20260620 22:52:18.744086 7979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:60791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:18.755335 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35447
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:18.832502 8309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:18.832706 8309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:18.832728 8309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:18.833858 8309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:18.833897 8309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:18.833923 8309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:18.835364 8309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:18.835583 8309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:18.835811 8309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:18.840310 8315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:18.840363 8314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:18.840310 8317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:18.840509 8309 server_base.cc:1034] running on GCE node
I20260620 22:52:18.840741 8309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:18.840965 8309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:18.842105 8309 hybrid_clock.cc:648] HybridClock initialized: now 1781995938842087 us; error 35 us; skew 500 ppm
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.844399 8309 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:18.844851 8309 webserver.cc:460] Webserver started at http://127.2.90.131:45281/ using document root <none> and password file <none>
I20260620 22:52:18.845062 8309 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:18.845109 8309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:18.845230 8309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:18.846196 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "b6ed2f18ab9b4250958c36f91547ff4f"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "cb0b9471177d622a18e6d71d67feb1fa"
server_key_iv: "85957fb7408b428d71b3c1704cd50793"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.846516 8309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "b6ed2f18ab9b4250958c36f91547ff4f"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "cb0b9471177d622a18e6d71d67feb1fa"
server_key_iv: "85957fb7408b428d71b3c1704cd50793"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.847661 8309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:52:18.848246 8324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.848412 8309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:18.848487 8309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "b6ed2f18ab9b4250958c36f91547ff4f"
format_stamp: "Formatted at 2026-06-20 22:52:18 on dist-test-slave-4mtw"
server_key: "cb0b9471177d622a18e6d71d67feb1fa"
server_key_iv: "85957fb7408b428d71b3c1704cd50793"
server_key_version: "encryptionkey@0"
I20260620 22:52:18.848554 8309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:18.876467 8309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:18.878566 8309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:18.878705 8309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:18.878916 8309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:18.879220 8309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:18.879252 8309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.879281 8309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:18.879298 8309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:18.884346 8309 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:36783
I20260620 22:52:18.884405 8437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:36783 every 8 connection(s)
I20260620 22:52:18.884694 8309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:18.889248 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8309
I20260620 22:52:18.889343 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:18.889523 2410 external_mini_cluster.cc:1428] Setting key e121be5b3d57480032ccfd374dd49bd0
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:18.893311 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.885335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:49077 (local address 127.2.90.190:35447)
0620 22:52:18.885393 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.885395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.885578 (+ 183us) server_negotiation.cc:407] Connection header received
0620 22:52:18.886482 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.886483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.886496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.886548 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.887170 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.887657 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.888621 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.888860 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.888915 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.890993 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.890994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.890994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.891001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.892252 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.892585 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.892586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.892586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.892632 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.892853 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.892854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.892854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.892952 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.893000 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.893196 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.893251 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:18.893486 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.885312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35447 (local address 127.2.90.131:49077)
0620 22:52:18.885460 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:18.885495 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:18.886371 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:18.886624 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:18.886628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:18.886864 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:18.887074 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:18.887079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.887739 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.887740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:18.888529 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:18.888531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.888934 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.888936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:18.889044 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.889506 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:18.889515 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:18.890921 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:18.892320 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:18.892325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:18.892327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:18.892528 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:18.892701 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:18.892702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:18.892702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:18.892804 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:18.892999 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:18.893002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:18.893108 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 22:52:18.893183 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:52:18.893290 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260620 22:52:18.893846 8438 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35447
I20260620 22:52:18.893932 8438 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:18.894093 8438 heartbeater.cc:511] Master 127.2.90.190:35447 requested a full tablet report, sending...
I20260620 22:52:18.894498 7979 ts_manager.cc:194] Registered new tserver with Master: b6ed2f18ab9b4250958c36f91547ff4f (127.2.90.131:36783)
I20260620 22:52:18.894879 7979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:49077
I20260620 22:52:18.901167 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/test-admin.keytab.
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.914947 2410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 22:52:18.916332 8445 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:18.915682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.2.90.129:41449)
0620 22:52:18.915843 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.915847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.915857 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:18.915880 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.915882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.915894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.915955 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.916268 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51852: BlockingRecv error: recv got EOF from 127.0.0.1:51852 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260620 22:52:18.916512 2410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.917376 2410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.918126 2410 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:52:18.924857 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.918553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.2.90.129:41449)
0620 22:52:18.918604 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.918606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.918644 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:52:18.918689 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.918690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.918699 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.918740 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.919075 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.919773 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.920455 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.920778 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.920827 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.922658 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.922666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.922667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.922673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.923991 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.924261 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.924263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.924263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.924311 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.924421 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.924421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.924422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.924525 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.924616 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.924670 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.924758 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 22:52:18.926281 8445 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:18.925879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.2.90.129:41449)
0620 22:52:18.925936 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.925937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.925944 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:18.925991 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.925991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.925999 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.926038 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.926241 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51874: BlockingRecv error: recv got EOF from 127.0.0.1:51874 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 22:52:18.926368 2410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.927168 2410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.927860 2410 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:52:18.933202 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.928280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.2.90.129:41449)
0620 22:52:18.928310 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.928312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.928331 (+ 19us) server_negotiation.cc:407] Connection header received
0620 22:52:18.928363 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.928364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.928371 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.928404 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.928720 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.929152 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.929746 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.929916 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.929957 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.931503 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.931504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.931504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.931509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.932478 (+ 969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.932754 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.932755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.932755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.932799 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.932912 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.932913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.932913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.932983 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.933037 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.933101 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.933153 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 22:52:18.934408 8445 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:18.933978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.2.90.129:41449)
0620 22:52:18.934023 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.934025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.934035 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:18.934087 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.934088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.934095 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.934144 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.934369 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51890: BlockingRecv error: recv got EOF from 127.0.0.1:51890 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 22:52:18.934464 2410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781995912840861-2410-0/krb5kdc/krb5cc' of type FILE
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.935225 2410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:18.935902 2410 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 22:52:18 dist-test-slave-4mtw krb5kdc[7934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:52:18.941452 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:18.936313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.2.90.129:41449)
0620 22:52:18.936350 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:18.936351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:18.936377 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:52:18.936432 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:18.936433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:18.936439 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:18.936481 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:18.936799 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.937216 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.937791 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:18.937954 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:18.937993 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:18.939594 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:18.939595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:18.939595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:18.939600 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:18.940730 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.940976 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.940977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.940977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.941017 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:18.941151 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:18.941152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:18.941152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:18.941218 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:18.941283 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:18.941346 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 22:52:18.941392 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:18.941998 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8041
I20260620 22:52:18.946235 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8175
I20260620 22:52:18.949874 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8309
I20260620 22:52:18.953254 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 7947
2026-06-20T22:52:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1707 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:19 dist-test-slave-4mtw krb5kdc[8458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:19 dist-test-slave-4mtw krb5kdc[8458](info): set up 2 sockets
Jun 20 22:52:19 dist-test-slave-4mtw krb5kdc[8458](info): commencing operation
krb5kdc: starting...
W20260620 22:52:19.896576 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:19 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab.
Entry for principal oryx/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab.
I20260620 22:52:20.003850 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35243
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab
--principal=oryx/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:20.080425 8471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:20.080618 8471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:20.080646 8471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:20.081764 8471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:20.081805 8471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:20.081823 8471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:20.081840 8471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:20.081854 8471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:20.083257 8471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab
--principal=oryx/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:20.083490 8471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:20.083683 8471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.087996 8477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.088011 8479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.087984 8476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.088282 8471 server_base.cc:1034] running on GCE node
I20260620 22:52:20.088507 8471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.088809 8471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.089948 8471 hybrid_clock.cc:648] HybridClock initialized: now 1781995940089934 us; error 29 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.092144 8471 init.cc:377] Logged in from keytab as oryx/127.2.90.190@KRBTEST.COM (short username oryx)
I20260620 22:52:20.092566 8471 webserver.cc:460] Webserver started at http://127.2.90.190:35319/ using document root <none> and password file <none>
I20260620 22:52:20.092798 8471 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.092845 8471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.092968 8471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:20.093923 8471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "6fb437d662bf4e8c90bc2a02c1b4af86"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "16789dc19f58ca305bbda72d4131ea88"
server_key_iv: "a6700dea0a38ff1e74e0691ff576d66f"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.094238 8471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "6fb437d662bf4e8c90bc2a02c1b4af86"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "16789dc19f58ca305bbda72d4131ea88"
server_key_iv: "a6700dea0a38ff1e74e0691ff576d66f"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.095378 8471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:20.096012 8486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.096172 8471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:20.096251 8471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "6fb437d662bf4e8c90bc2a02c1b4af86"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "16789dc19f58ca305bbda72d4131ea88"
server_key_iv: "a6700dea0a38ff1e74e0691ff576d66f"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.096318 8471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.107848 8471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:20.109562 8471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:20.109686 8471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:20.113271 8471 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35243
I20260620 22:52:20.113309 8538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35243 every 8 connection(s)
I20260620 22:52:20.113607 8471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:20.114495 8539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:20.116535 8539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Bootstrap starting.
I20260620 22:52:20.117077 8539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:20.117301 8539 log.cc:824] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:20.117692 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8471
I20260620 22:52:20.117801 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:20.117925 8539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: No bootstrap required, opened a new log
I20260620 22:52:20.117988 2410 external_mini_cluster.cc:1428] Setting key 3c52b7ebb572e01a71978d076b1bc0a2
I20260620 22:52:20.119874 8539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } }
I20260620 22:52:20.120039 8539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:20.120074 8539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb437d662bf4e8c90bc2a02c1b4af86, State: Initialized, Role: FOLLOWER
I20260620 22:52:20.120186 8539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } }
I20260620 22:52:20.120268 8539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:20.120294 8539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:20.120339 8539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:20.121346 8539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } }
I20260620 22:52:20.121450 8539 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb437d662bf4e8c90bc2a02c1b4af86; no voters:
I20260620 22:52:20.121588 8539 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:20.121613 8544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:20.121743 8544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb437d662bf4e8c90bc2a02c1b4af86, State: Running, Role: LEADER
I20260620 22:52:20.121801 8539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:20.121837 8544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } }
I20260620 22:52:20.122133 8544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb437d662bf4e8c90bc2a02c1b4af86. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } }
I20260620 22:52:20.122215 8544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:20.122154 8545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } }
I20260620 22:52:20.122330 8545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:20.122400 8550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:20.122892 8550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:20.124639 8550 catalog_manager.cc:1347] Generated new cluster ID: 204e316b818946a6934eb4439f9aff02
I20260620 22:52:20.124701 8550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:20.129757 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.118431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.2.90.190:35243)
0620 22:52:20.118623 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.118632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.118663 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:52:20.119849 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.119857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.119921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.120064 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.120732 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.122060 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.123150 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.123653 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.123756 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.126288 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.126293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.126293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.126304 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.128159 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.128519 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.128522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.128523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.128649 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.128880 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.128881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.128881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.129187 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.129344 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.129412 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.129547 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260620 22:52:20.140875 8550 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:20.141340 8550 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:20.147434 8550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Generated new TSK 0
I20260620 22:52:20.147619 8550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab.
Entry for principal oryx/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab.
I20260620 22:52:20.175181 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35243
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:20.252991 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:20.253182 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:20.253211 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:20.254300 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:20.254341 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:20.254375 8565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:20.255755 8565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:20.255972 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:20.256176 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.260825 8570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.260838 8571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.260901 8565 server_base.cc:1034] running on GCE node
W20260620 22:52:20.260869 8573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.261224 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.261472 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.262607 8565 hybrid_clock.cc:648] HybridClock initialized: now 1781995940262596 us; error 29 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.264926 8565 init.cc:377] Logged in from keytab as oryx/127.2.90.129@KRBTEST.COM (short username oryx)
I20260620 22:52:20.265321 8565 webserver.cc:460] Webserver started at http://127.2.90.129:41439/ using document root <none> and password file <none>
I20260620 22:52:20.265542 8565 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.265589 8565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.265708 8565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:20.266696 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "a72a62afdf3f4af5a2adb6599aaa701f"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "87fba0904551e45e37b8ff2995d41558"
server_key_iv: "f77a572eb59737709c4ea313bff3133b"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.267020 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "a72a62afdf3f4af5a2adb6599aaa701f"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "87fba0904551e45e37b8ff2995d41558"
server_key_iv: "f77a572eb59737709c4ea313bff3133b"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.268198 8565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:20.268844 8580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.269012 8565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:20.269089 8565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "a72a62afdf3f4af5a2adb6599aaa701f"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "87fba0904551e45e37b8ff2995d41558"
server_key_iv: "f77a572eb59737709c4ea313bff3133b"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.269158 8565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.287448 8565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:20.289213 8565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:20.289474 8565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:20.289688 8565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:20.289981 8565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:20.290014 8565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.290043 8565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:20.290060 8565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.295496 8565 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37037
I20260620 22:52:20.295529 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37037 every 8 connection(s)
I20260620 22:52:20.295826 8565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:20.300024 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8565
I20260620 22:52:20.300140 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:20.300347 2410 external_mini_cluster.cc:1428] Setting key add18aba6f7bce741d92d503bffe3f72
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:20.304750 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.296442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:60305 (local address 127.2.90.190:35243)
0620 22:52:20.296522 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.296525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.296705 (+ 180us) server_negotiation.cc:407] Connection header received
0620 22:52:20.297177 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.297178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.297191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.297243 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.298106 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.298606 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.299556 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.299769 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.299831 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.302305 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.302306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.302307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.302315 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.303467 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.303819 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.303820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.303821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.303875 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.304091 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.304092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.304092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.304196 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.304251 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.304509 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.304595 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 22:52:20.304862 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.296414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:60305)
0620 22:52:20.296556 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.296591 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.297087 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.297295 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.297298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.297835 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.298020 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.298024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.298696 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.298697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.299493 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.299494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.299869 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.299870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.300021 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.300704 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.300714 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.302208 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.303543 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.303549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.303551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.303751 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.303931 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.303932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.303932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.304035 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.304258 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.304265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.304392 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.304475 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.304624 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260620 22:52:20.305718 8694 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35243
I20260620 22:52:20.305812 8694 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:20.305985 8694 heartbeater.cc:511] Master 127.2.90.190:35243 requested a full tablet report, sending...
I20260620 22:52:20.306581 8503 ts_manager.cc:194] Registered new tserver with Master: a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129:37037)
I20260620 22:52:20.307442 8503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:60305
WARNING: no policy specified for oryx/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab.
Entry for principal oryx/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab.
I20260620 22:52:20.323107 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35243
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:20.401324 8699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:20.401511 8699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:20.401532 8699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:20.402590 8699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:20.402623 8699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:20.402647 8699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:20.403980 8699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:20.404263 8699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:20.404598 8699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.409338 8704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.409353 8705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.409636 8699 server_base.cc:1034] running on GCE node
W20260620 22:52:20.409343 8707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.409879 8699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.410115 8699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.411252 8699 hybrid_clock.cc:648] HybridClock initialized: now 1781995940411239 us; error 24 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.413547 8699 init.cc:377] Logged in from keytab as oryx/127.2.90.130@KRBTEST.COM (short username oryx)
I20260620 22:52:20.413962 8699 webserver.cc:460] Webserver started at http://127.2.90.130:45195/ using document root <none> and password file <none>
I20260620 22:52:20.414134 8699 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.414170 8699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.414258 8699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:20.415232 8699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "9a04f613507e4819be7aa4faaab6681b"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "6d14e5e18426279a247cb2b05475ca5f"
server_key_iv: "e92a3d0956cf4410a3209b97efda2217"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.415527 8699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "9a04f613507e4819be7aa4faaab6681b"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "6d14e5e18426279a247cb2b05475ca5f"
server_key_iv: "e92a3d0956cf4410a3209b97efda2217"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.416651 8699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:20.417349 8714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.417533 8699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:20.417609 8699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "9a04f613507e4819be7aa4faaab6681b"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "6d14e5e18426279a247cb2b05475ca5f"
server_key_iv: "e92a3d0956cf4410a3209b97efda2217"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.417662 8699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.429813 8699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:20.431363 8699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:20.431473 8699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:20.431644 8699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:20.431871 8699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:20.431898 8699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.431917 8699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:20.431928 8699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.437142 8699 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:35623
I20260620 22:52:20.437212 8827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:35623 every 8 connection(s)
I20260620 22:52:20.437500 8699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:20.445569 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.438152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:34771 (local address 127.2.90.190:35243)
0620 22:52:20.438210 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.438211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.438400 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:52:20.438853 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.438854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.438864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.438924 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.439786 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.440272 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.441234 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.441422 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.441471 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.443357 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.443358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.443359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.443365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.444435 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.444864 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.444865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.444865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.444913 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.445093 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.445093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.445093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.445181 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.445222 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.445442 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.445508 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:20.445730 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.438112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:34771)
0620 22:52:20.438278 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.438318 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.438775 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.438989 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.438992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.439535 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.439719 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.439724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.440334 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.440334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.441169 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.441171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.441479 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.441480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.441571 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.442028 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.442036 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.443278 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.444533 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.444538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.444540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.444807 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.444961 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.444961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.444962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.445050 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.445219 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.445221 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.445350 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.445427 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.445536 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260620 22:52:20.446146 8828 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35243
I20260620 22:52:20.446226 8828 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:20.446393 8828 heartbeater.cc:511] Master 127.2.90.190:35243 requested a full tablet report, sending...
I20260620 22:52:20.446718 8503 ts_manager.cc:194] Registered new tserver with Master: 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:20.447067 8503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:34771
I20260620 22:52:20.447575 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8699
I20260620 22:52:20.447661 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:20.447840 2410 external_mini_cluster.cc:1428] Setting key 473ecfcbae0c0db00e56989a7e5fe075
WARNING: no policy specified for oryx/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab.
Entry for principal oryx/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab.
I20260620 22:52:20.469286 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35243
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:20.547008 8833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:20.547190 8833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:20.547211 8833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:20.548316 8833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:20.548357 8833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:20.548381 8833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:20.549820 8833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:20.550031 8833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:20.550233 8833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.554740 8839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.554852 8838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.554751 8841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.554922 8833 server_base.cc:1034] running on GCE node
I20260620 22:52:20.555159 8833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.555377 8833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.556515 8833 hybrid_clock.cc:648] HybridClock initialized: now 1781995940556506 us; error 27 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.558813 8833 init.cc:377] Logged in from keytab as oryx/127.2.90.131@KRBTEST.COM (short username oryx)
I20260620 22:52:20.559180 8833 webserver.cc:460] Webserver started at http://127.2.90.131:37221/ using document root <none> and password file <none>
I20260620 22:52:20.559355 8833 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.559391 8833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.559480 8833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:20.560415 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "f98740a4e4084f34abc9dc5019e9e8b8"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "9953266b45bf60eb2643c16f1a835b57"
server_key_iv: "80911c8f9b3c464f447988870016971c"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.560734 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "f98740a4e4084f34abc9dc5019e9e8b8"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "9953266b45bf60eb2643c16f1a835b57"
server_key_iv: "80911c8f9b3c464f447988870016971c"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.561846 8833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:20.562458 8848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.562600 8833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:20.562678 8833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "f98740a4e4084f34abc9dc5019e9e8b8"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "9953266b45bf60eb2643c16f1a835b57"
server_key_iv: "80911c8f9b3c464f447988870016971c"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.562736 8833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.577266 8833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:20.578928 8833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:20.579032 8833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:20.579209 8833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:20.579456 8833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:20.579485 8833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.579507 8833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:20.579522 8833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.584738 8833 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:36525
I20260620 22:52:20.584774 8961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:36525 every 8 connection(s)
I20260620 22:52:20.585071 8833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:20.593093 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8833
I20260620 22:52:20.593171 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.585745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:46743 (local address 127.2.90.190:35243)
0620 22:52:20.585793 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.585795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.585979 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:52:20.586438 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.586439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.586447 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.586493 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.587348 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.587838 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.588804 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.588989 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.589035 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.591037 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.591038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.591038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.591044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.592074 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.592411 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.592412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.592412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.592464 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.592639 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.592640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.592640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.592754 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.592801 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.593043 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.593102 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:20.593196 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:20.593379 2410 external_mini_cluster.cc:1428] Setting key b3790c416f954ac10c69eb4530a9717d
I20260620 22:52:20.593430 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.585724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:46743)
0620 22:52:20.585868 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.585904 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.586359 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.586550 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.586553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.587092 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.587282 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.587287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.587902 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.587902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.588738 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.588742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.589042 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.589043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.589137 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.589592 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.589600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.590958 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.592136 (+ 1178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.592141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.592142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.592357 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.592509 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.592510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.592510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.592600 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.592797 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.592800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.592920 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.593032 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.593195 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 22:52:20.593902 8962 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35243
I20260620 22:52:20.593997 8962 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:20.594172 8962 heartbeater.cc:511] Master 127.2.90.190:35243 requested a full tablet report, sending...
I20260620 22:52:20.594537 8503 ts_manager.cc:194] Registered new tserver with Master: f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525)
I20260620 22:52:20.594898 8503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:46743
I20260620 22:52:20.605010 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.90.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.189.keytab.
Entry for principal oryx/127.2.90.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.189.keytab.
I20260620 22:52:20.628065 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:34037
--webserver_interface=127.2.90.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.189.keytab
--principal=oryx/127.2.90.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35243,127.2.90.189:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:20.706236 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:20.706436 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:20.706456 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:20.707557 8967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:20.707597 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:20.707609 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:20.707631 8967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:20.707653 8967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:20.709105 8967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35243,127.2.90.189:34037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.189.keytab
--principal=oryx/127.2.90.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.90.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
W20260620 22:52:20.709221 8967 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.90.190:35243,127.2.90.189:34037'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 22:52:20.709357 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:20.709546 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.713943 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.714073 8967 server_base.cc:1034] running on GCE node
W20260620 22:52:20.713971 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.713953 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.714392 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.714653 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.715802 8967 hybrid_clock.cc:648] HybridClock initialized: now 1781995940715785 us; error 31 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.718024 8967 init.cc:377] Logged in from keytab as oryx/127.2.90.189@KRBTEST.COM (short username oryx)
I20260620 22:52:20.718444 8967 webserver.cc:460] Webserver started at http://127.2.90.189:34157/ using document root <none> and password file <none>
I20260620 22:52:20.718634 8967 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.718679 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.718803 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:20.719748 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/instance:
uuid: "5928a3b908dd40909a3a1882e237b0ec"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "f45a9fccb306f6862006ebea6c9df474"
server_key_iv: "910c0ce3aad0b4248c862bb36616bb49"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.720067 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal/instance:
uuid: "5928a3b908dd40909a3a1882e237b0ec"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "f45a9fccb306f6862006ebea6c9df474"
server_key_iv: "910c0ce3aad0b4248c862bb36616bb49"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.721275 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:20.721922 8982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.722115 8967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:20.722214 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
uuid: "5928a3b908dd40909a3a1882e237b0ec"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "f45a9fccb306f6862006ebea6c9df474"
server_key_iv: "910c0ce3aad0b4248c862bb36616bb49"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.722332 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.733646 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:20.735183 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:20.735314 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:20.738833 8967 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.189:34037
I20260620 22:52:20.738862 9034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.189:34037 every 8 connection(s)
I20260620 22:52:20.739154 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
I20260620 22:52:20.740005 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:20.741453 9035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:52:20.742362 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 8967
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.189@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
I20260620 22:52:20.750134 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.742104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.2.90.190:35243)
0620 22:52:20.742193 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.742196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.742329 (+ 133us) server_negotiation.cc:407] Connection header received
0620 22:52:20.742864 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.742865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.742878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.742937 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.743555 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.744069 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.745135 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.745413 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.745464 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.747538 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.747539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.747540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.747546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.748779 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.749194 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.749195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.749195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.749251 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.749466 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.749466 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.749467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.749645 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.749717 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.749990 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.750067 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 22:52:20.750298 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.742024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:35884)
0620 22:52:20.742199 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.742242 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.742715 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.743012 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.743016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.743329 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.743470 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.743474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.744160 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.744160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.745063 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.745065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.745479 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.745480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.745601 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.746102 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.746109 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.747466 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.748847 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.748852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.748854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.749129 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.749303 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.749304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.749304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.749398 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.749718 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.749723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.749861 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.749959 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.750085 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260620 22:52:20.750643 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.742779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.2.90.189:34037)
0620 22:52:20.742924 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.742928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.742944 (+ 16us) server_negotiation.cc:407] Connection header received
0620 22:52:20.743006 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.743008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.743026 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.743109 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.743625 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.744597 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.745375 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.745730 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.745805 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.748031 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.748037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.748037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.748042 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.749481 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.749796 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.749798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.749799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.749855 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.750031 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.750032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.750032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.750175 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.750301 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.750494 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.750557 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260620 22:52:20.750890 9035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.189@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
I20260620 22:52:20.755774 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.751108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52826)
0620 22:52:20.751137 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.751146 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.751178 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.751307 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.751307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.751401 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.751434 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.751435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.752017 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.752017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.752483 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.752484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.752811 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.752812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.752849 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.753115 (+ 266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.753118 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.754068 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.755121 (+ 1053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.755122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.755122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.755259 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.755376 (+ 117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.755377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.755377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.755423 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.755570 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.755570 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.755599 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.755642 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.755704 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 22:52:20.755829 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.751139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.2.90.189:34037)
0620 22:52:20.751178 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.751180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.751188 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:20.751224 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.751225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.751234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.751302 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.751487 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.751951 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.752538 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.752766 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.752811 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.754133 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.754134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.754134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.754139 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.755063 (+ 924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.755303 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.755303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.755304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.755342 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.755455 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.755455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.755455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.755528 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.755566 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.755657 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.755741 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":19}
I20260620 22:52:20.756924 9035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Bootstrap starting.
I20260620 22:52:20.757645 9035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:20.757879 9035 log.cc:824] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:20.758477 9035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: No bootstrap required, opened a new log
I20260620 22:52:20.759781 9035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } }
I20260620 22:52:20.759891 9035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:20.759922 9035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5928a3b908dd40909a3a1882e237b0ec, State: Initialized, Role: FOLLOWER
I20260620 22:52:20.760037 9035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } }
I20260620 22:52:20.760304 9035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:20.760270 9041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } } }
I20260620 22:52:20.760361 9041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:52:20.761720 9052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:52:20.761772 9052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:52:20.763207 8967 master_runner.cc:428] Detected that this master 5928a3b908dd40909a3a1882e237b0ec is joining an existing cluster
I20260620 22:52:20.763248 8967 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.90.189:34037
I20260620 22:52:20.769783 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.764293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52832)
0620 22:52:20.764485 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.764492 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.764545 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.764673 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.764673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.764877 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.764933 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.764935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.765708 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.765709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.766254 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.766256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.766618 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.766619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.766685 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.767102 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.767106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.767498 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.768854 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.768855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.768856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.769017 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.769213 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.769214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.769214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.769264 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.769480 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.769480 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.769516 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.769575 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.769668 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260620 22:52:20.769790 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.764441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.2.90.189:34037)
0620 22:52:20.764471 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.764472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.764555 (+ 83us) server_negotiation.cc:407] Connection header received
0620 22:52:20.764594 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.764595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.764603 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.764650 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.765008 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.765649 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.766319 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.766513 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.766565 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.767581 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.767582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.767582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.767587 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.768792 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.769090 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.769092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.769092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.769149 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.769323 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.769324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.769324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.769427 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.769480 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.769601 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.769715 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":17}
I20260620 22:52:20.771361 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.764091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.2.90.190:35243)
0620 22:52:20.764126 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.764127 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.764248 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:52:20.764299 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.764300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.764310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.764353 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.764766 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.765470 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.766554 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.766816 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.766876 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.767649 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.767650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.767651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.767659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.769829 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.770224 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.770225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.770225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.770284 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.770488 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.770489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.770489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.770598 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.770648 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.771231 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.771304 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:52:20.771422 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.764055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:35886)
0620 22:52:20.764184 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.764193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.764247 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.764355 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.764356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.764523 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.764614 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.764615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.765533 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.765534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.766468 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.766471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.766881 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.766882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.766950 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.767224 (+ 274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.767228 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.767566 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.769914 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.769915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.769915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.770108 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.770356 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.770357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.770357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.770419 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.770660 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.770660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.770700 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.771181 (+ 481us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.771323 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.190@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
I20260620 22:52:20.779842 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.773341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.2.90.189:34037)
0620 22:52:20.773416 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.773417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.773487 (+ 70us) server_negotiation.cc:407] Connection header received
0620 22:52:20.773579 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.773579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.773586 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.773626 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.774066 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.774470 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.775520 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.775679 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.775716 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.777711 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.777712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.777713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.777719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.778860 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.779256 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.779257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.779257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.779297 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.779465 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.779465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.779466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.779543 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.779584 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.779744 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.779789 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 22:52:20.779942 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.773278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52840)
0620 22:52:20.773396 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.773441 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.773512 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.773632 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.773635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.773945 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.773999 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.774001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.774533 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.774534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.775454 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.775456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.775735 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.775736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.775806 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.776193 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.776198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.777593 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.778936 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.778942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.778945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.779193 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.779350 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.779351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.779351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.779413 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.779594 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.779599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.779646 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.779727 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.779833 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260620 22:52:20.780458 8503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.90.189:34037: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec"
member_type: NON_VOTER
last_known_addr {
host: "127.2.90.189"
port: 34037
}
attrs {
promote: true
}
}
dest_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86"
cas_config_opid_index: -1
I20260620 22:52:20.780776 8503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } }
I20260620 22:52:20.781217 9061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5928a3b908dd40909a3a1882e237b0ec (127.2.90.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } }
I20260620 22:52:20.781461 9009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:20.782025 9061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.90.189:34037
I20260620 22:52:20.782186 9009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Refusing update from remote peer 6fb437d662bf4e8c90bc2a02c1b4af86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 22:52:20.782212 8967 master.cc:537] Master@127.2.90.189:34037 shutting down...
I20260620 22:52:20.782176 9062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } }
I20260620 22:52:20.782243 9062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:20.782645 9062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:52:20.784019 8487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 22:52:20.784485 8487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 -> Peer 5928a3b908dd40909a3a1882e237b0ec (127.2.90.189:34037): Couldn't send request to peer 5928a3b908dd40909a3a1882e237b0ec. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:52:20.784853 8967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:20.784924 8967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:20.784976 8967 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: stopping tablet replica
I20260620 22:52:20.797235 8967 master.cc:552] Master@127.2.90.189:34037 shutdown complete.
I20260620 22:52:20.797276 8967 master_runner.cc:305] Clearing existing system tablet
I20260620 22:52:20.797382 8967 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.797425 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.797947 8967 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:20.798384 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.798511 8967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.798561 8967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.798583 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
uuid: "5928a3b908dd40909a3a1882e237b0ec"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "f45a9fccb306f6862006ebea6c9df474"
server_key_iv: "910c0ce3aad0b4248c862bb36616bb49"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.799281 8967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:20.800740 8967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 22:52:20.800801 8967 log.cc:1196] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 22:52:20.800985 8967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Deleting consensus metadata
I20260620 22:52:20.801080 8967 master_runner.cc:315] Copying system tablet from 127.2.90.190:35243
I20260620 22:52:20.801633 8967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: tablet copy: Beginning tablet copy session from remote peer at address 127.2.90.190:35243
I20260620 22:52:20.807009 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.801963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.2.90.190:35243)
0620 22:52:20.802011 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.802013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.802027 (+ 14us) server_negotiation.cc:407] Connection header received
0620 22:52:20.802105 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.802106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.802115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.802161 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.802448 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.802956 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.803659 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.803857 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.803909 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.804802 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.804803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.804803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.804810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.806041 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.806339 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.806340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.806341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.806392 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.806564 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.806564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.806565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.806663 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.806717 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.806851 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.806916 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:52:20.807031 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.801878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:35902)
0620 22:52:20.801975 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:20.801985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:20.802047 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:20.802163 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:20.802163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:20.802323 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:20.802376 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.802377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.803032 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.803033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.803583 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:20.803585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.803917 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.803918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:20.803982 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.804439 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:20.804442 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:20.804731 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:20.806119 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.806120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.806120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.806277 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.806458 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:20.806459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:20.806459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:20.806513 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:20.806718 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:20.806719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:20.806762 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:52:20.806822 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:52:20.806931 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":66,"threads_started":1}
I20260620 22:52:20.807325 8523 tablet_copy_service.cc:140] P 6fb437d662bf4e8c90bc2a02c1b4af86: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5928a3b908dd40909a3a1882e237b0ec ({username='oryx', principal='oryx/127.2.90.189@KRBTEST.COM'} at 127.0.0.1:35902)
I20260620 22:52:20.807406 8523 tablet_copy_service.cc:161] P 6fb437d662bf4e8c90bc2a02c1b4af86: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5928a3b908dd40909a3a1882e237b0ec at {username='oryx', principal='oryx/127.2.90.189@KRBTEST.COM'} at 127.0.0.1:35902: session id = 5928a3b908dd40909a3a1882e237b0ec-00000000000000000000000000000000
I20260620 22:52:20.807976 8523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:52:20.808516 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 22:52:20.809370 8967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: tablet copy: Starting download of 0 data blocks...
I20260620 22:52:20.809485 8967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: tablet copy: Starting download of 1 WAL segments...
I20260620 22:52:20.810220 8967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:52:20.810930 8523 tablet_copy_service.cc:342] P 6fb437d662bf4e8c90bc2a02c1b4af86: Request end of tablet copy session 5928a3b908dd40909a3a1882e237b0ec-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.90.189@KRBTEST.COM'} at 127.0.0.1:35902
I20260620 22:52:20.810986 8523 tablet_copy_service.cc:433] P 6fb437d662bf4e8c90bc2a02c1b4af86: ending tablet copy session 5928a3b908dd40909a3a1882e237b0ec-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5928a3b908dd40909a3a1882e237b0ec
I20260620 22:52:20.813035 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.814105 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.814217 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.814430 8967 server_base.cc:1034] running on GCE node
W20260620 22:52:20.814105 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.814606 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.814728 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.815838 8967 hybrid_clock.cc:648] HybridClock initialized: now 1781995940815829 us; error 25 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.817032 8967 init.cc:377] Logged in from keytab as oryx/127.2.90.189@KRBTEST.COM (short username oryx)
I20260620 22:52:20.817250 8967 webserver.cc:460] Webserver started at http://127.2.90.189:33483/ using document root <none> and password file <none>
I20260620 22:52:20.817349 8967 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.817373 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.817884 8967 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.818339 9083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.818440 8967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.818481 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
uuid: "5928a3b908dd40909a3a1882e237b0ec"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "f45a9fccb306f6862006ebea6c9df474"
server_key_iv: "910c0ce3aad0b4248c862bb36616bb49"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.818533 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.837718 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:20.839118 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:20.841868 8967 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.189:34037
I20260620 22:52:20.842064 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.189:34037 every 8 connection(s)
I20260620 22:52:20.842190 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
I20260620 22:52:20.842837 9136 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:52:20.843226 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Bootstrap starting.
I20260620 22:52:20.845424 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:52:20.845611 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: Bootstrap complete.
I20260620 22:52:20.845801 9136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } }
I20260620 22:52:20.845865 9136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5928a3b908dd40909a3a1882e237b0ec, State: Initialized, Role: LEARNER
I20260620 22:52:20.845890 9136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } }
I20260620 22:52:20.846031 9136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:20.846051 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } }
I20260620 22:52:20.846163 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: LEARNER
I20260620 22:52:20.847309 9150 catalog_manager.cc:1259] Loaded cluster ID: 204e316b818946a6934eb4439f9aff02
I20260620 22:52:20.847354 9150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: loading cluster ID for follower catalog manager: success
I20260620 22:52:20.848223 9150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: acquiring CA information for follower catalog manager: success
I20260620 22:52:20.848472 9150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:52:20.858402 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:20.852015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.2.90.189:34037)
0620 22:52:20.852150 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:20.852152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:20.852158 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:52:20.852176 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:20.852177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:20.852185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:20.852245 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:20.852948 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.853564 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.854594 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:20.854773 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:20.854814 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:20.856124 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:20.856125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:20.856126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:20.856131 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:20.857386 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.857764 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.857765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.857765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.857823 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:20.858005 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:20.858005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:20.858006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:20.858097 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:20.858163 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:20.858290 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 22:52:20.858342 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.2.90.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.188.keytab.
Entry for principal oryx/127.2.90.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.188.keytab.
I20260620 22:52:20.881417 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:39207
--webserver_interface=127.2.90.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.188.keytab
--principal=oryx/127.2.90.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:20.960177 9154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:20.960390 9154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:20.960421 9154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:20.961623 9154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:20.961671 9154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:20.961690 9154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:20.961706 9154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:20.961722 9154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:20.963227 9154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.188.keytab
--principal=oryx/127.2.90.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:39207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.90.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:20.963465 9154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:20.963668 9154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:20.968238 9160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.968269 9162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:20.968255 9159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:20.968482 9154 server_base.cc:1034] running on GCE node
I20260620 22:52:20.968747 9154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:20.969028 9154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:20.970170 9154 hybrid_clock.cc:648] HybridClock initialized: now 1781995940970153 us; error 30 us; skew 500 ppm
Jun 20 22:52:20 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:20.972635 9154 init.cc:377] Logged in from keytab as oryx/127.2.90.188@KRBTEST.COM (short username oryx)
I20260620 22:52:20.973079 9154 webserver.cc:460] Webserver started at http://127.2.90.188:38583/ using document root <none> and password file <none>
I20260620 22:52:20.973276 9154 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:20.973325 9154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:20.973456 9154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:20.974486 9154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/instance:
uuid: "3a53343101d145abad9bc883aa2f347c"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "7a4c7f6d20318a099dd14e395d9ed0b2"
server_key_iv: "28185522fa9041e5a40587efce320896"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.974812 9154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal/instance:
uuid: "3a53343101d145abad9bc883aa2f347c"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "7a4c7f6d20318a099dd14e395d9ed0b2"
server_key_iv: "28185522fa9041e5a40587efce320896"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.976082 9154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:52:20.976800 9169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:20.976958 9154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:20.977041 9154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
uuid: "3a53343101d145abad9bc883aa2f347c"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "7a4c7f6d20318a099dd14e395d9ed0b2"
server_key_iv: "28185522fa9041e5a40587efce320896"
server_key_version: "encryptionkey@0"
I20260620 22:52:20.977111 9154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:20.999235 9154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:21.001400 9154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:21.001542 9154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:21.005702 9154 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.188:39207
I20260620 22:52:21.005745 9221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.188:39207 every 8 connection(s)
I20260620 22:52:21.006064 9154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
I20260620 22:52:21.006994 9222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:21.008488 9222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.188@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:21.015930 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 9154
I20260620 22:52:21.016819 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.009391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.2.90.190:35243)
0620 22:52:21.009449 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.009451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.009561 (+ 110us) server_negotiation.cc:407] Connection header received
0620 22:52:21.010104 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.010105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.010114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.010156 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.010711 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.011189 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.012094 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.012279 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.012320 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.014264 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.014265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.014266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.014271 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.015482 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.015806 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.015807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.015808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.015862 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.016107 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.016108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.016108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.016239 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.016310 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.016626 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.016745 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:52:21.017035 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.009259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:35912)
0620 22:52:21.009441 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.009481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.010001 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.010224 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.010227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.010463 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.010650 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.010655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.011248 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.011248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.012034 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.012036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.012334 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.012335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.012429 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.012874 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.012883 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.014174 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.015545 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.015550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.015551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.015746 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.015921 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.015922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.015922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.016037 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.016309 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.016316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.016488 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.016589 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.016754 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260620 22:52:21.017520 9222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.188@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.90.188@KRBTEST.COM
I20260620 22:52:21.023679 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.017742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52874)
0620 22:52:21.017780 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.017790 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.017824 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.017973 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.017973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.018071 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.018111 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.018112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.018768 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.018769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.019286 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.019287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.019681 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.019682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.019743 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.020069 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.020072 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.021240 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.022710 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.022711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.022712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.022969 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.023142 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.023142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.023143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.023199 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.023394 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.023395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.023442 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.023516 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.023587 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 22:52:21.023679 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.017817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.2.90.189:34037)
0620 22:52:21.017860 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.017862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.017869 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:21.017890 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.017891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.017903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.017960 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.018181 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.018681 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.019372 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.019622 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.019675 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.021308 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.021309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.021310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.021315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.022626 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.023037 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.023038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.023038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.023086 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.023254 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.023255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.023255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.023341 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.023384 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.023533 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.023595 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:52:21.024154 9222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:52:21.024840 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.016394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.2.90.188:39207)
0620 22:52:21.016544 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.016548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.016566 (+ 18us) server_negotiation.cc:407] Connection header received
0620 22:52:21.016586 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.016589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.016601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.016672 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.017314 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.018238 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.018917 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.019221 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.019279 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.021974 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.021980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.021980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.021986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.023559 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.023905 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.023908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.023908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.023971 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.024260 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.024261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.024261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.024416 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.024550 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.024616 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.024748 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.188@KRBTEST.COM for oryx/127.2.90.188@KRBTEST.COM
I20260620 22:52:21.029824 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.024325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.0.0.1:37292)
0620 22:52:21.024467 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.024476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.024513 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.024614 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.024614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.024775 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.024799 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.024800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.025474 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.025475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.025945 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.025946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.026262 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.026262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.026306 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.026574 (+ 268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.026578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.027518 (+ 940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.029153 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.029154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.029155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.029293 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.029433 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.029433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.029434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.029475 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.029624 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.029624 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.029652 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.029693 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.029750 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 22:52:21.029914 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.024374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.2.90.188:39207)
0620 22:52:21.024492 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.024493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.024502 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:52:21.024553 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.024554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.024561 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.024614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.024858 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.025423 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.026012 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.026207 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.026255 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.027590 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.027591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.027591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.027596 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.029092 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.029347 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.029347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.029348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.029389 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.029508 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.029508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.029508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.029586 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.029624 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.029708 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.029788 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260620 22:52:21.030967 9222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Bootstrap starting.
I20260620 22:52:21.031476 9222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:21.031699 9222 log.cc:824] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:21.032299 9222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: No bootstrap required, opened a new log
I20260620 22:52:21.033692 9222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } }
I20260620 22:52:21.033805 9222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:21.033825 9222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a53343101d145abad9bc883aa2f347c, State: Initialized, Role: FOLLOWER
I20260620 22:52:21.033927 9222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } }
I20260620 22:52:21.034180 9222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:21.034152 9228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } } }
I20260620 22:52:21.034236 9228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:52:21.035744 9239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:52:21.035801 9239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:52:21.037894 9154 master_runner.cc:428] Detected that this master 3a53343101d145abad9bc883aa2f347c is joining an existing cluster
I20260620 22:52:21.037933 9154 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.90.188:39207
I20260620 22:52:21.045356 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.038813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.2.90.190:35243)
0620 22:52:21.038852 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.038854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.039236 (+ 382us) server_negotiation.cc:407] Connection header received
0620 22:52:21.039263 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.039264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.039274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.039317 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.039780 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.040499 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.041678 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.041962 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.041994 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.043128 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.043129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.043130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.043136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.044339 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.044630 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.044631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.044631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.044716 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.044884 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.044885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.044885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.044992 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.045047 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.045193 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.045262 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:21.045369 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.038904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:35918)
0620 22:52:21.039175 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.039182 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.039226 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.039356 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.039357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.039637 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.039705 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.039706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.040573 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.040574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.041599 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.041601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.042054 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.042054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.042120 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.042755 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.042758 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.043041 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.044413 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.044414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.044415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.044566 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.044771 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.044772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.044772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.044830 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.045046 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.045047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.045103 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.045159 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.045263 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":101,"thread_start_us":133,"threads_started":1}
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995940, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.190@KRBTEST.COM for oryx/127.2.90.188@KRBTEST.COM
I20260620 22:52:21.064486 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.047502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.0.0.1:37316)
0620 22:52:21.047547 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.047556 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.047588 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.047692 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.047693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.047847 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.047887 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.047889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.048442 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.048442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.049258 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.049260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.049575 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.049576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.049635 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.049944 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.049948 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.051129 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.063375 (+ 12246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.063376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.063586 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.063785 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.063786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.063842 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.064158 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.064159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.064204 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.064263 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.064365 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:52:21.064524 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.038793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.2.90.188:39207)
0620 22:52:21.038827 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.038829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.039163 (+ 334us) server_negotiation.cc:407] Connection header received
0620 22:52:21.039188 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.039189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.039197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.039250 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.039791 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.040506 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.041631 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.041916 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.041953 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.061843 (+ 19890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.061843 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.061844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.061849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.063486 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063812 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.063812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.063813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.063854 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.064033 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.064033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.064033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.064158 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.064191 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.064358 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.064431 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:52:21.064528 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.038949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.2.90.189:34037)
0620 22:52:21.038993 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.038995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.039003 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:21.039078 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.039079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.039089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.039121 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.039700 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.040464 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.041653 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.041946 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.041982 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.061735 (+ 19753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.061737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.061737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.061745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.063168 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063773 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.063774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.063774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.063827 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.064027 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.064028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.064028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.064141 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.064182 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.064365 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.064445 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:21.064819 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.038776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52890)
0620 22:52:21.038958 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.038968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.039015 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.039290 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.039291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.039556 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.039615 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.039616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.040550 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.040550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.041558 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.041560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.042053 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.042054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.042120 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.061098 (+ 18978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.061104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.061618 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.063264 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.063266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.063693 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.063894 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.063895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.063958 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.064225 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.064225 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.064264 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.064595 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.064718 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":69,"thread_start_us":71,"threads_started":1}
I20260620 22:52:21.065209 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.038854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.0.0.1:37308)
0620 22:52:21.039096 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.039103 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.039152 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.039297 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.039297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.039633 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.039713 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.039715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.040587 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.040587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.041546 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.041549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.042032 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.042033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.042088 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.061333 (+ 19245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.061337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.061764 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.063545 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.063546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.063751 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.063922 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.063923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.063974 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.064224 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.064225 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.064269 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.064622 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.065121 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":516,"thread_start_us":56,"threads_started":1}
I20260620 22:52:21.065536 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.047537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.2.90.188:39207)
0620 22:52:21.047571 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.047572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.047580 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:21.047640 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.047641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.047647 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.047688 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.047944 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.048372 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.049322 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.049516 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.049561 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.051205 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.051206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.051207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.051213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.063306 (+ 12093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063672 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.063673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.063673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.063723 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.063899 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.063899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.063900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.064074 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.064130 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.065140 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.065213 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:52:21.065891 8503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.90.188:39207: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3a53343101d145abad9bc883aa2f347c"
member_type: NON_VOTER
last_known_addr {
host: "127.2.90.188"
port: 39207
}
attrs {
promote: true
}
}
dest_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86"
cas_config_opid_index: 5
I20260620 22:52:21.066121 8503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } }
I20260620 22:52:21.066445 9061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3a53343101d145abad9bc883aa2f347c (127.2.90.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.066639 9196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:21.067302 9061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.90.188:39207
I20260620 22:52:21.067369 9061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.067437 9061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.067516 9196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Refusing update from remote peer 6fb437d662bf4e8c90bc2a02c1b4af86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 22:52:21.067773 9062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:21.067847 9154 master.cc:537] Master@127.2.90.188:39207 shutting down...
W20260620 22:52:21.068899 8490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 -> Peer 3a53343101d145abad9bc883aa2f347c (127.2.90.188:39207): Couldn't send request to peer 3a53343101d145abad9bc883aa2f347c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:52:21.069880 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.066477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52898)
0620 22:52:21.066608 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.066617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.066659 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.066846 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.066847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.066849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.066886 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.066887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.067708 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.067709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.068885 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.068889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.069629 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.069630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.069700 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.069725 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.069776 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.069815 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 22:52:21.070175 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.066554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.2.90.189:34037)
0620 22:52:21.066585 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.066587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.066703 (+ 116us) server_negotiation.cc:407] Connection header received
0620 22:52:21.066722 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.066723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.066743 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.066791 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.066961 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.067619 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.068985 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.069533 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.069599 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.069675 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.070078 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.070116 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:52:21.070535 9110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 LEARNER]: Refusing update from remote peer 6fb437d662bf4e8c90bc2a02c1b4af86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 22:52:21.070796 9062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 22:52:21.070875 9154 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:21.070969 9154 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:21.071018 9154 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: stopping tablet replica
I20260620 22:52:21.071341 9110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3a53343101d145abad9bc883aa2f347c (127.2.90.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.072448 9249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.072593 9249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: LEARNER
I20260620 22:52:21.072644 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb437d662bf4e8c90bc2a02c1b4af86. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: NON_VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: true } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.072731 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: LEARNER
I20260620 22:52:21.072942 9062 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: attempting to promote NON_VOTER 5928a3b908dd40909a3a1882e237b0ec to VOTER
I20260620 22:52:21.073134 9062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } }
I20260620 22:52:21.073575 9110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 LEARNER]: Refusing update from remote peer 6fb437d662bf4e8c90bc2a02c1b4af86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 22:52:21.073741 9062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 22:52:21.073884 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.073719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.0.0.1:37326)
0620 22:52:21.073819 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.073858 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:39207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 22:52:21.073940 8490 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 -> Peer 3a53343101d145abad9bc883aa2f347c (127.2.90.188:39207): Couldn't send request to peer 3a53343101d145abad9bc883aa2f347c. Status: Network error: Client connection negotiation failed: client connection to 127.2.90.188:39207: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 22:52:21.074553 9061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5928a3b908dd40909a3a1882e237b0ec (127.2.90.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.074687 9110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5928a3b908dd40909a3a1882e237b0ec (127.2.90.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.075237 9061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.075316 9061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.075276 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.075455 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.075690 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.075783 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:52:21.085165 9154 master.cc:552] Master@127.2.90.188:39207 shutdown complete.
I20260620 22:52:21.085206 9154 master_runner.cc:305] Clearing existing system tablet
I20260620 22:52:21.085321 9154 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:21.085359 9154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:21.085907 9154 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:21.086315 9252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.086462 9154 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:21.086513 9154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.086535 9154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
uuid: "3a53343101d145abad9bc883aa2f347c"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "7a4c7f6d20318a099dd14e395d9ed0b2"
server_key_iv: "28185522fa9041e5a40587efce320896"
server_key_version: "encryptionkey@0"
I20260620 22:52:21.087126 9154 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:21.088342 9154 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 22:52:21.088393 9154 log.cc:1196] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 22:52:21.088559 9154 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Deleting consensus metadata
I20260620 22:52:21.088656 9154 master_runner.cc:315] Copying system tablet from 127.2.90.190:35243
I20260620 22:52:21.089244 9154 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: tablet copy: Beginning tablet copy session from remote peer at address 127.2.90.190:35243
I20260620 22:52:21.094677 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.089570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.2.90.190:35243)
0620 22:52:21.089618 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.089620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.089744 (+ 124us) server_negotiation.cc:407] Connection header received
0620 22:52:21.089796 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.089797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.089806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.089849 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.090140 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.090617 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.091294 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.091486 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.091537 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.092404 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.092405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.092405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.092412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.093703 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.094004 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.094005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.094006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.094061 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.094220 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.094221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.094221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.094325 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.094378 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.094523 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.094587 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:52:21.094687 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.089508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:35930)
0620 22:52:21.089674 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.089684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.089741 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.089852 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.089853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.090012 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.090071 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.090073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.090686 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.090686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.091221 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.091222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.091545 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.091546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.091607 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.092043 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.092047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.092325 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.093775 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.093776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.093777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.093942 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.094112 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.094113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.094113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.094169 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.094378 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.094378 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.094431 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.094488 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.094587 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260620 22:52:21.094869 8523 tablet_copy_service.cc:140] P 6fb437d662bf4e8c90bc2a02c1b4af86: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3a53343101d145abad9bc883aa2f347c ({username='oryx', principal='oryx/127.2.90.188@KRBTEST.COM'} at 127.0.0.1:35930)
I20260620 22:52:21.094921 8523 tablet_copy_service.cc:161] P 6fb437d662bf4e8c90bc2a02c1b4af86: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3a53343101d145abad9bc883aa2f347c at {username='oryx', principal='oryx/127.2.90.188@KRBTEST.COM'} at 127.0.0.1:35930: session id = 3a53343101d145abad9bc883aa2f347c-00000000000000000000000000000000
I20260620 22:52:21.095304 8523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:52:21.095706 9154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 22:52:21.096526 9154 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: tablet copy: Starting download of 0 data blocks...
I20260620 22:52:21.096633 9154 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: tablet copy: Starting download of 1 WAL segments...
I20260620 22:52:21.097263 9154 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:52:21.098078 8523 tablet_copy_service.cc:342] P 6fb437d662bf4e8c90bc2a02c1b4af86: Request end of tablet copy session 3a53343101d145abad9bc883aa2f347c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.90.188@KRBTEST.COM'} at 127.0.0.1:35930
I20260620 22:52:21.098131 8523 tablet_copy_service.cc:433] P 6fb437d662bf4e8c90bc2a02c1b4af86: ending tablet copy session 3a53343101d145abad9bc883aa2f347c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3a53343101d145abad9bc883aa2f347c
I20260620 22:52:21.100235 9154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:21.101293 9259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.101373 9260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.101508 9262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:21.101662 9154 server_base.cc:1034] running on GCE node
I20260620 22:52:21.101751 9154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:21.101873 9154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:21.103009 9154 hybrid_clock.cc:648] HybridClock initialized: now 1781995941102998 us; error 22 us; skew 500 ppm
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:21.104251 9154 init.cc:377] Logged in from keytab as oryx/127.2.90.188@KRBTEST.COM (short username oryx)
I20260620 22:52:21.104523 9154 webserver.cc:460] Webserver started at http://127.2.90.188:36577/ using document root <none> and password file <none>
I20260620 22:52:21.104637 9154 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:21.104672 9154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:21.105376 9154 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:21.105866 9269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.105971 9154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:21.106012 9154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
uuid: "3a53343101d145abad9bc883aa2f347c"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "7a4c7f6d20318a099dd14e395d9ed0b2"
server_key_iv: "28185522fa9041e5a40587efce320896"
server_key_version: "encryptionkey@0"
I20260620 22:52:21.106065 9154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:21.111471 9154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:21.112900 9154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:21.115823 9154 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.188:39207
I20260620 22:52:21.116019 9321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.188:39207 every 8 connection(s)
I20260620 22:52:21.116135 9154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
I20260620 22:52:21.117012 9322 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:52:21.117523 9322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Bootstrap starting.
I20260620 22:52:21.120445 9322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:52:21.120672 9322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Bootstrap complete.
I20260620 22:52:21.120888 9322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } }
I20260620 22:52:21.120977 9322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a53343101d145abad9bc883aa2f347c, State: Initialized, Role: LEARNER
I20260620 22:52:21.121012 9322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } }
I20260620 22:52:21.121184 9322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:21.121189 9325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.121304 9325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: This master's current role is: LEARNER
I20260620 22:52:21.122637 9336 catalog_manager.cc:1259] Loaded cluster ID: 204e316b818946a6934eb4439f9aff02
I20260620 22:52:21.122682 9336 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: loading cluster ID for follower catalog manager: success
I20260620 22:52:21.123972 9336 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: acquiring CA information for follower catalog manager: success
I20260620 22:52:21.124179 9336 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:52:21.131695 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.126200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.2.90.188:39207)
0620 22:52:21.126297 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.126299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.126306 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:21.126326 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.126327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.126335 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.126387 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.126853 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.127471 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.128128 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.128328 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.128379 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.129484 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.129486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.129486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.129492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.130737 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.131036 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.131037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.131037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.131091 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.131248 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.131249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.131249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.131349 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.131438 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.131528 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.131597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260620 22:52:21.132155 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8565
I20260620 22:52:21.136606 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:37037
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:35243
--webserver_port=41439
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:21.234190 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:21.234405 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:21.234429 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:21.235566 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:21.235605 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:21.235630 9338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:21.237085 9338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:37037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=41439
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:21.237291 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:21.237522 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:21.242331 9344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.242511 9346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.242571 9343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:21.242964 9338 server_base.cc:1034] running on GCE node
I20260620 22:52:21.243137 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:21.243321 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:21.244465 9338 hybrid_clock.cc:648] HybridClock initialized: now 1781995941244450 us; error 36 us; skew 500 ppm
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:21.246855 9338 init.cc:377] Logged in from keytab as oryx/127.2.90.129@KRBTEST.COM (short username oryx)
I20260620 22:52:21.247262 9338 webserver.cc:460] Webserver started at http://127.2.90.129:41439/ using document root <none> and password file <none>
I20260620 22:52:21.247466 9338 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:21.247522 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:21.248940 9338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:21.249457 9353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.249603 9338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.249680 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "a72a62afdf3f4af5a2adb6599aaa701f"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "87fba0904551e45e37b8ff2995d41558"
server_key_iv: "f77a572eb59737709c4ea313bff3133b"
server_key_version: "encryptionkey@0"
I20260620 22:52:21.249994 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:21.258641 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:21.260186 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:21.260311 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:21.260509 9338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:21.260811 9338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:21.260844 9338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.260871 9338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:21.260888 9338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.266995 9338 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37037
I20260620 22:52:21.267055 9466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37037 every 8 connection(s)
I20260620 22:52:21.267355 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:21.272887 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 9338
I20260620 22:52:21.273043 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8699
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
I20260620 22:52:21.279124 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.268231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:46723 (local address 127.2.90.190:35243)
0620 22:52:21.268274 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.268275 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.268284 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:52:21.269349 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.269350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.269362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.269391 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.270174 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.270920 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272331 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.272615 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272671 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.276237 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.276239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.276239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.276247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.277733 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.278235 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.278236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.278236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.278287 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.278511 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.278512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.278512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.278623 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.278678 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.278956 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.279041 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:52:21.279129 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.268030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:47977 (local address 127.2.90.189:34037)
0620 22:52:21.268080 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.268082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.268442 (+ 360us) server_negotiation.cc:407] Connection header received
0620 22:52:21.269330 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.269332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.269343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.269414 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.270142 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.270912 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272751 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.273033 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.273094 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.276444 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.276445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.276445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.276451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.277806 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.278205 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.278206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.278207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.278266 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.278531 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.278532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.278532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.278675 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.278715 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.278960 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.279039 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:52:21.279126 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:35623
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:35243
--webserver_port=45195
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:52:21.279587 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.267968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.2.90.129:47977)
0620 22:52:21.268365 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.268374 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.269237 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.269496 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.269500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.269874 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.270058 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.270062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.271006 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.271007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.272191 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.272194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.273482 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.273483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.273544 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.274460 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.274464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.276115 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.277864 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.277864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.277865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.278140 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.278328 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.278329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.278329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.278450 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.278755 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.278756 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.278817 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.279219 (+ 402us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.279379 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":61,"thread_start_us":72,"threads_started":1}
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.188@KRBTEST.COM
I20260620 22:52:21.280200 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.267968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:46723)
0620 22:52:21.268148 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.268187 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.269226 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.269495 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.269500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.269867 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.270058 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.270064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.271007 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.271007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.272191 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.272194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.272704 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.272764 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.273732 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.273741 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.276105 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.277814 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.277820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.277822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.278144 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.278364 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.278365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.278366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.278447 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.278681 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.278687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.278814 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.280006 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.280101 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":50,"thread_start_us":82,"threads_started":1}
I20260620 22:52:21.280432 9469 heartbeater.cc:347] Connected to a master server at 127.2.90.189:34037
I20260620 22:52:21.280532 9469 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.280740 9469 heartbeater.cc:511] Master 127.2.90.189:34037 requested a full tablet report, sending...
I20260620 22:52:21.280764 9468 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35243
I20260620 22:52:21.280808 9468 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.280874 9468 heartbeater.cc:511] Master 127.2.90.190:35243 requested a full tablet report, sending...
I20260620 22:52:21.281132 8503 ts_manager.cc:194] Re-registered known tserver with Master: a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129:37037)
I20260620 22:52:21.281206 9100 ts_manager.cc:194] Registered new tserver with Master: a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129:37037)
I20260620 22:52:21.281569 8503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:46723
I20260620 22:52:21.283686 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.268215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:46101 (local address 127.2.90.188:39207)
0620 22:52:21.268264 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.268266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.268384 (+ 118us) server_negotiation.cc:407] Connection header received
0620 22:52:21.269349 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.269350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.269363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.269391 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.271003 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.271485 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272257 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.272539 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272602 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.281155 (+ 8553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.281157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.281157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.281165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.282720 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.283026 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.283027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.283027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.283078 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.283235 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.283236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.283236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.283336 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.283390 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.283523 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.283602 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:21.283697 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.267968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.2.90.129:46101)
0620 22:52:21.268308 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.268316 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.269240 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.269497 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.269500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.269877 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.270058 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.270064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.271554 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.271555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.272164 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.272166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.272611 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.272612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.272759 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.278911 (+ 6152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.278915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.281062 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.282800 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.282801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.282801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.282961 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.283132 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.283133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.283133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.283187 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.283393 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.283394 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.283433 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.283492 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.283594 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":4,"thread_start_us":76,"threads_started":1}
I20260620 22:52:21.283977 9467 heartbeater.cc:347] Connected to a master server at 127.2.90.188:39207
I20260620 22:52:21.284018 9467 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.284063 9467 heartbeater.cc:511] Master 127.2.90.188:39207 requested a full tablet report, sending...
I20260620 22:52:21.284377 9286 ts_manager.cc:194] Registered new tserver with Master: a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129:37037)
W20260620 22:52:21.359916 9474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:21.360102 9474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:21.360131 9474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:21.361279 9474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:21.361318 9474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:21.361361 9474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:21.362787 9474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:35623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=45195
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:21.363000 9474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:21.363199 9474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:21.367936 9480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:21.367957 9474 server_base.cc:1034] running on GCE node
W20260620 22:52:21.367988 9482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.367942 9479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:21.368378 9474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:21.368568 9474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:21.369732 9474 hybrid_clock.cc:648] HybridClock initialized: now 1781995941369725 us; error 28 us; skew 500 ppm
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:21.372053 9474 init.cc:377] Logged in from keytab as oryx/127.2.90.130@KRBTEST.COM (short username oryx)
I20260620 22:52:21.372452 9474 webserver.cc:460] Webserver started at http://127.2.90.130:45195/ using document root <none> and password file <none>
I20260620 22:52:21.372649 9474 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:21.372723 9474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:21.373932 9474 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:21.374444 9489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.374593 9474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:21.374655 9474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "9a04f613507e4819be7aa4faaab6681b"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "6d14e5e18426279a247cb2b05475ca5f"
server_key_iv: "e92a3d0956cf4410a3209b97efda2217"
server_key_version: "encryptionkey@0"
I20260620 22:52:21.374979 9474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:21.396596 9474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:21.398301 9474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:21.398417 9474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:21.398633 9474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:21.398922 9474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:21.398955 9474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.398984 9474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:21.399008 9474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.404296 9474 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:35623
I20260620 22:52:21.404376 9602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:35623 every 8 connection(s)
I20260620 22:52:21.404632 9474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.188@KRBTEST.COM
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
I20260620 22:52:21.414463 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.405368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:53573 (local address 127.2.90.188:39207)
0620 22:52:21.405460 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.405462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.405636 (+ 174us) server_negotiation.cc:407] Connection header received
0620 22:52:21.406209 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.406210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.406221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.406250 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.407182 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.407664 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.408834 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.409131 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409192 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.411733 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.411735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.411735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.411742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.413102 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413488 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.413489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.413489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.413550 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413833 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.413834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.413835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.413945 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.414008 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.414285 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.414365 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 22:52:21.414635 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 9474
I20260620 22:52:21.414788 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8833
I20260620 22:52:21.415047 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.405319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:56515 (local address 127.2.90.190:35243)
0620 22:52:21.405424 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.405427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.405636 (+ 209us) server_negotiation.cc:407] Connection header received
0620 22:52:21.406209 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.406210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.406221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.406250 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.407740 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.408192 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409073 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.409255 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409306 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.411424 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.411425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.411425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.411432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.413021 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413510 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.413511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.413512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.413564 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413804 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.413805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.413805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.413944 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.414011 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.414893 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.414975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 22:52:21.415093 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.405311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:56515)
0620 22:52:21.405581 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.405590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.406111 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.406325 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.406329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.406843 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.407069 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.407075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.408269 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.408270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.409004 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.409006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.409313 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.409388 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.409866 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.409876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.411328 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.413078 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.413086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.413416 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.413634 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.413635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.413739 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.414722 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.414723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.414765 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.414834 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.414926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260620 22:52:21.415182 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.405310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.2.90.130:53573)
0620 22:52:21.405520 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.405560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.406095 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.406325 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.406329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.406843 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.407068 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.407075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.407749 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.407750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.408768 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.408770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.409250 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.409383 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.410065 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.410068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.411651 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.413182 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.413183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.413418 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.413618 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.413619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.413743 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.414011 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.414016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.414158 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.414261 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.414401 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":10,"thread_start_us":98,"threads_started":1}
I20260620 22:52:21.415612 9603 heartbeater.cc:347] Connected to a master server at 127.2.90.188:39207
I20260620 22:52:21.415702 9603 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.415882 9603 heartbeater.cc:511] Master 127.2.90.188:39207 requested a full tablet report, sending...
I20260620 22:52:21.416267 9286 ts_manager.cc:194] Registered new tserver with Master: 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:21.416742 9604 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35243
I20260620 22:52:21.416793 9604 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.416857 9604 heartbeater.cc:511] Master 127.2.90.190:35243 requested a full tablet report, sending...
I20260620 22:52:21.417117 8503 ts_manager.cc:194] Re-registered known tserver with Master: 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:21.417595 8503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:56515
I20260620 22:52:21.420110 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:36525
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:35243
--webserver_port=37221
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:52:21.420243 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.405579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:37651 (local address 127.2.90.189:34037)
0620 22:52:21.405621 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.405623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.405761 (+ 138us) server_negotiation.cc:407] Connection header received
0620 22:52:21.406179 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.406181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.406191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.406238 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.408260 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.408727 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409617 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.409822 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409881 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.412277 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.412278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.412278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.412286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.413785 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.414095 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.414096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.414096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.414140 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.414309 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.414309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.414309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.414423 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.414483 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.420086 (+ 5603us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.420155 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:21.420423 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.405316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.2.90.130:37651)
0620 22:52:21.405687 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.405696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.406104 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.406614 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.406616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.406838 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.407068 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.407073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.408793 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.408794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.409559 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.409561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.409879 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.409880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.409935 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.410259 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.410263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.412181 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.413855 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.413856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.413856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.413994 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.414196 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.414197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.414197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.414256 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.419919 (+ 5663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.419920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.419961 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.420034 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.420147 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":3,"spinlock_wait_cycles":6528,"thread_start_us":105,"threads_started":1}
I20260620 22:52:21.420737 9605 heartbeater.cc:347] Connected to a master server at 127.2.90.189:34037
I20260620 22:52:21.420784 9605 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.420840 9605 heartbeater.cc:511] Master 127.2.90.189:34037 requested a full tablet report, sending...
I20260620 22:52:21.421103 9100 ts_manager.cc:194] Registered new tserver with Master: 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:21.458881 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.456076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.2.90.188:39207)
0620 22:52:21.456131 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.456132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.456141 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:52:21.456198 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.456199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.456216 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.456267 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.456418 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.456924 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.458223 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.458586 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.458636 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.458679 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.458796 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.458826 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:52:21.458915 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.456014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.0.0.1:37354)
0620 22:52:21.456069 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.456083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.456144 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.456290 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.456291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.456300 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.456350 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.456353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.456993 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.456994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.458147 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.458150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.458663 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.458663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.458727 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.458732 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.458787 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.458831 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:52:21.459219 9296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 22:52:21.459352 9325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb437d662bf4e8c90bc2a02c1b4af86. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.459466 9325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: This master's current role is: LEARNER
W20260620 22:52:21.498438 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:21.498646 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:21.498670 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:21.499822 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:21.499861 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:21.499889 9610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:21.501336 9610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:36525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=37221
--tserver_master_addrs=127.2.90.190:35243,127.2.90.189:34037,127.2.90.188:39207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:21.501554 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:21.501744 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:21.506134 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.506132 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:21.506239 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:21.506407 9610 server_base.cc:1034] running on GCE node
I20260620 22:52:21.506668 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:21.506856 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:21.507979 9610 hybrid_clock.cc:648] HybridClock initialized: now 1781995941507968 us; error 27 us; skew 500 ppm
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:21.510205 9610 init.cc:377] Logged in from keytab as oryx/127.2.90.131@KRBTEST.COM (short username oryx)
I20260620 22:52:21.510588 9610 webserver.cc:460] Webserver started at http://127.2.90.131:37221/ using document root <none> and password file <none>
I20260620 22:52:21.510774 9610 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:21.510833 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:21.512064 9610 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:21.512583 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.512758 9610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.512835 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "f98740a4e4084f34abc9dc5019e9e8b8"
format_stamp: "Formatted at 2026-06-20 22:52:20 on dist-test-slave-4mtw"
server_key: "9953266b45bf60eb2643c16f1a835b57"
server_key_iv: "80911c8f9b3c464f447988870016971c"
server_key_version: "encryptionkey@0"
I20260620 22:52:21.513149 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:21.524915 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:21.526449 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:21.526569 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:21.526751 9610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:21.527010 9610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:21.527041 9610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.527068 9610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:21.527093 9610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:21.532829 9610 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:36525
I20260620 22:52:21.532882 9738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:36525 every 8 connection(s)
I20260620 22:52:21.533181 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:21.534463 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 9610
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.189@KRBTEST.COM
Jun 20 22:52:21 dist-test-slave-4mtw krb5kdc[8458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995941, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.188@KRBTEST.COM
I20260620 22:52:21.544602 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.534137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37575 (local address 127.2.90.189:34037)
0620 22:52:21.534177 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.534179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.534799 (+ 620us) server_negotiation.cc:407] Connection header received
0620 22:52:21.535203 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.535205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.535216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.535245 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.535655 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.536435 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.537911 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538212 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.538250 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.541087 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.541088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.541089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.541096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.543069 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543580 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.543581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.543581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.543638 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543900 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.543901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.543902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.544030 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.544067 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.544433 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.544509 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:21.545140 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.534082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.2.90.131:37575)
0620 22:52:21.534747 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.534755 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.535117 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.535302 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.535303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.535434 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.535557 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.535563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.536527 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.536528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.537813 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.537815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538321 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.538321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.538429 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.539012 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.539022 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.540979 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.543148 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.543157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.543497 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.543706 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.543707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.543824 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.544110 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.544116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.544267 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.544745 (+ 478us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.544912 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":77,"thread_start_us":62,"threads_started":1}
I20260620 22:52:21.545332 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.533781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:49103)
0620 22:52:21.533970 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.534009 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.534584 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.534906 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.534907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.535281 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.535557 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.535563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.536527 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.536528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.537814 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.537815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538321 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.538322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.538431 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.539157 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.539162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.541713 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.543837 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543837 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.543838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.544375 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.544594 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.544595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.544595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.544645 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.544935 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.544935 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.544973 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.545139 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.545232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":129,"spinlock_wait_cycles":6912,"thread_start_us":97,"threads_started":1}
I20260620 22:52:21.545557 9741 heartbeater.cc:347] Connected to a master server at 127.2.90.189:34037
I20260620 22:52:21.545653 9741 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.545821 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.533783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.2.90.131:49633)
0620 22:52:21.534179 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.534188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.534587 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.534822 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.534826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.535285 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:21.535558 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.535566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.536456 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.536457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.537818 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.537820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538294 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.538295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.538430 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.539736 (+ 1306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:21.539741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:21.541420 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:21.543247 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.543249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.543524 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.544051 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:21.544052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:21.544052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:21.544908 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:21.545533 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:21.545533 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:21.545568 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.545625 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.545727 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":69,"thread_start_us":107,"threads_started":1}
I20260620 22:52:21.545856 9741 heartbeater.cc:511] Master 127.2.90.189:34037 requested a full tablet report, sending...
I20260620 22:52:21.545873 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.534018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:49633 (local address 127.2.90.188:39207)
0620 22:52:21.534090 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.534093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.534234 (+ 141us) server_negotiation.cc:407] Connection header received
0620 22:52:21.534685 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.534686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.534697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.534749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.535659 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.536334 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.537902 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538182 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.538216 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.541492 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.541493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.541493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.541500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.543062 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.543905 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.543906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.543906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.543968 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.544990 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.544991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.544991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.545114 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.545152 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.545709 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.545772 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:21.546030 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.533779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:49103 (local address 127.2.90.190:35243)
0620 22:52:21.533836 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.533839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.534087 (+ 248us) server_negotiation.cc:407] Connection header received
0620 22:52:21.534807 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.534808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.534819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.534865 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.535656 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.536444 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.537911 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538212 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.538250 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.541803 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.541804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.541805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.541811 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.543760 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.544472 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.544473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.544473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.544527 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.544740 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.544740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.544741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.544853 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.544892 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.545079 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.545951 (+ 872us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:21.546274 9100 ts_manager.cc:194] Registered new tserver with Master: f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525)
I20260620 22:52:21.546607 9739 heartbeater.cc:347] Connected to a master server at 127.2.90.188:39207
I20260620 22:52:21.546653 9739 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.546705 9739 heartbeater.cc:511] Master 127.2.90.188:39207 requested a full tablet report, sending...
I20260620 22:52:21.546958 9286 ts_manager.cc:194] Registered new tserver with Master: f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525)
I20260620 22:52:21.547123 9740 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35243
I20260620 22:52:21.547173 9740 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:21.547304 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.535505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.2.90.188:39207)
0620 22:52:21.536136 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.536138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.536274 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:52:21.536294 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.536295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.536305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.536349 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.537544 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.538235 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.539774 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.540075 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.540136 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.542429 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.542430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.542431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.542437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.545962 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.546342 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.546343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.546343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.546392 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.546558 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.546559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.546559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.546669 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.546749 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.547148 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.547221 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":55,"threads_started":1}
I20260620 22:52:21.547328 9740 heartbeater.cc:511] Master 127.2.90.190:35243 requested a full tablet report, sending...
I20260620 22:52:21.547458 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.535885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.2.90.190:35243)
0620 22:52:21.536751 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.536752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.536972 (+ 220us) server_negotiation.cc:407] Connection header received
0620 22:52:21.536994 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.536994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.537006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.537063 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.538870 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.539618 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.541150 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.541436 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.541489 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.543679 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.543680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.543680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.543686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.545327 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.545659 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.545660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.545660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.545714 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.546778 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.546779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.546779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.546899 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.547006 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.547269 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.547345 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":79,"threads_started":1}
I20260620 22:52:21.547518 8503 ts_manager.cc:194] Re-registered known tserver with Master: f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525)
I20260620 22:52:21.547849 8503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:49103
I20260620 22:52:21.549307 8503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:21.551302 8503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:21.551988 9062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: NON_VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: true } } }
I20260620 22:52:21.552080 9062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.553895 9061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: attempting to promote NON_VOTER 3a53343101d145abad9bc883aa2f347c to VOTER
I20260620 22:52:21.554004 9061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } }
I20260620 22:52:21.554682 9295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 LEARNER]: Refusing update from remote peer 6fb437d662bf4e8c90bc2a02c1b4af86: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 22:52:21.554888 9062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 22:52:21.555099 9110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Refusing update from remote peer 6fb437d662bf4e8c90bc2a02c1b4af86: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 22:52:21.555527 9062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 22:52:21.555785 9250 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3a53343101d145abad9bc883aa2f347c (127.2.90.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.555902 9295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3a53343101d145abad9bc883aa2f347c (127.2.90.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.556376 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.541111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.2.90.189:34037)
0620 22:52:21.542792 (+ 1681us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.542794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.542907 (+ 113us) server_negotiation.cc:407] Connection header received
0620 22:52:21.543998 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.543999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.544010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.544044 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.549357 (+ 5313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.549813 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.550816 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.550997 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.551039 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.553209 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.553211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.553211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.553219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.554929 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.555271 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.555272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.555272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.555326 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.555503 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.555505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.555505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.555737 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.556185 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.556222 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.556296 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1662,"thread_start_us":51,"threads_started":1}
I20260620 22:52:21.556769 9110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3a53343101d145abad9bc883aa2f347c (127.2.90.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.556814 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.556893 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.556939 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.556914 9325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.556977 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.556981 9325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:52:21.557034 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.557077 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:21.557540 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:21.557598 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:52:21.561470 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.557649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37037 (local address 127.0.0.1:53576)
0620 22:52:21.557690 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.557699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.557732 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.558207 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.558207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.558209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.558261 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.558262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.559167 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.559168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.560349 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.560351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.561262 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.561263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.561314 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.561318 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.561373 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.561410 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":50}
I20260620 22:52:21.561487 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.557728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.2.90.129:37037)
0620 22:52:21.557865 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.557869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.557879 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:21.557916 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.557918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.557937 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.558036 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.558381 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.559091 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.560437 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.561175 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.561248 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.561320 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.561388 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.561429 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260620 22:52:21.562006 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.557808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:36525 (local address 127.0.0.1:49016)
0620 22:52:21.557913 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.557922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.557964 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.558383 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.558384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.558386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.558435 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.558437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.559220 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.559220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.560361 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.560363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.561784 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.561785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.561845 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.561849 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.561903 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.561941 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":69,"threads_started":1}
I20260620 22:52:21.562040 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.558085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.2.90.131:36525)
0620 22:52:21.558238 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.558242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.558252 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:21.558286 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.558288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.558304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.558373 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.558489 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.559136 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.560435 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.561179 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.561248 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.561324 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.561930 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.561969 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260620 22:52:21.562175 9401 tablet_service.cc:1467] Processing CreateTablet for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07 (DEFAULT_TABLE table=test-table [id=3220d0d1e86e40baa2ece70d0796d044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:21.562305 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.557970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35623 (local address 127.0.0.1:55742)
0620 22:52:21.559284 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.559293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.559342 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.559492 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.559492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.559494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.559531 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.559533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.560465 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.560466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.561191 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.561193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.562106 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.562106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.562163 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.562166 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.562210 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.562244 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"thread_start_us":64,"threads_started":1}
I20260620 22:52:21.562453 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.558066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.2.90.130:35623)
0620 22:52:21.558477 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.558481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.559338 (+ 857us) server_negotiation.cc:407] Connection header received
0620 22:52:21.559401 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.559404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.559419 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.559493 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.559581 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.560393 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.561274 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.561950 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.562003 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.562075 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.562238 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.562285 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":79,"threads_started":1}
I20260620 22:52:21.562482 9401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:21.562974 9537 tablet_service.cc:1467] Processing CreateTablet for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07 (DEFAULT_TABLE table=test-table [id=3220d0d1e86e40baa2ece70d0796d044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:21.563201 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:21.565351 9673 tablet_service.cc:1467] Processing CreateTablet for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07 (DEFAULT_TABLE table=test-table [id=3220d0d1e86e40baa2ece70d0796d044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:21.565542 9768 tablet_bootstrap.cc:492] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Bootstrap starting.
I20260620 22:52:21.565682 9673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:21.566222 9768 tablet_bootstrap.cc:654] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:21.566493 9768 log.cc:824] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:21.567268 9768 tablet_bootstrap.cc:492] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: No bootstrap required, opened a new log
I20260620 22:52:21.567329 9768 ts_tablet_manager.cc:1397] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:21.568290 9771 tablet_bootstrap.cc:492] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Bootstrap starting.
I20260620 22:52:21.568465 9769 tablet_bootstrap.cc:492] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Bootstrap starting.
I20260620 22:52:21.568928 9771 tablet_bootstrap.cc:654] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:21.569026 9769 tablet_bootstrap.cc:654] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:21.569190 9768 raft_consensus.cc:348] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.569336 9768 raft_consensus.cc:374] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:21.569378 9768 raft_consensus.cc:729] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72a62afdf3f4af5a2adb6599aaa701f, State: Initialized, Role: FOLLOWER
I20260620 22:52:21.569494 9768 consensus_queue.cc:260] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.569738 9768 ts_tablet_manager.cc:1428] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:52:21.569715 9769 log.cc:824] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:21.569864 9468 heartbeater.cc:503] Master 127.2.90.190:35243 was elected leader, sending a full tablet report...
I20260620 22:52:21.569715 9771 log.cc:824] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:21.570806 9769 tablet_bootstrap.cc:492] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: No bootstrap required, opened a new log
I20260620 22:52:21.570880 9769 ts_tablet_manager.cc:1397] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:21.572829 9769 raft_consensus.cc:348] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.572969 9769 raft_consensus.cc:374] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:21.573014 9769 raft_consensus.cc:729] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a04f613507e4819be7aa4faaab6681b, State: Initialized, Role: FOLLOWER
I20260620 22:52:21.573125 9769 consensus_queue.cc:260] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.573241 9771 tablet_bootstrap.cc:492] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: No bootstrap required, opened a new log
I20260620 22:52:21.573308 9771 ts_tablet_manager.cc:1397] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 22:52:21.573374 9769 ts_tablet_manager.cc:1428] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:52:21.573483 9604 heartbeater.cc:503] Master 127.2.90.190:35243 was elected leader, sending a full tablet report...
I20260620 22:52:21.575299 9771 raft_consensus.cc:348] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.575423 9771 raft_consensus.cc:374] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:21.575443 9771 raft_consensus.cc:729] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98740a4e4084f34abc9dc5019e9e8b8, State: Initialized, Role: FOLLOWER
I20260620 22:52:21.575526 9771 consensus_queue.cc:260] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.575723 9771 ts_tablet_manager.cc:1428] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:21.575768 9740 heartbeater.cc:503] Master 127.2.90.190:35243 was elected leader, sending a full tablet report...
W20260620 22:52:21.655366 9606 tablet.cc:2369] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:21.722467 9772 raft_consensus.cc:482] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:21.722579 9772 raft_consensus.cc:504] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.722863 9772 leader_election.cc:288] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525), 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:21.725620 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.723161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:59143 (local address 127.2.90.131:36525)
0620 22:52:21.723217 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.723219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.723232 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:52:21.723261 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.723262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.723274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.723328 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.723478 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.724015 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.724972 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.725320 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.725379 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.725387 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.725522 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.725554 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:21.725620 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.723102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:36525 (local address 127.2.90.129:59143)
0620 22:52:21.723143 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.723154 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.723199 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.723348 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.723349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.723351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.723418 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.723420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.724082 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.724083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.724890 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.724892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.725386 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.725387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.725461 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.725464 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.725509 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.725546 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 22:52:21.725672 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.723348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:44435 (local address 127.2.90.130:35623)
0620 22:52:21.723491 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.723493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.723498 (+ 5us) server_negotiation.cc:407] Connection header received
0620 22:52:21.723517 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.723517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.723526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.723560 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:21.723702 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.724142 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.725035 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.725379 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.725440 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.725448 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.725581 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.725611 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 22:52:21.725704 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.723198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35623 (local address 127.2.90.129:44435)
0620 22:52:21.723251 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:21.723262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:21.723318 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:21.723580 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:21.723581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:21.723583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:21.723642 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.723643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.724216 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.724217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.724955 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:21.724957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.725455 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.725456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:21.725523 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.725525 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:21.725569 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:52:21.725609 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:52:21.725895 9693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b0479133a4f4ba4d1e1c6c5c9ca07" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" is_pre_election: true
I20260620 22:52:21.726012 9693 raft_consensus.cc:2454] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 0.
I20260620 22:52:21.726208 9354 leader_election.cc:302] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72a62afdf3f4af5a2adb6599aaa701f, f98740a4e4084f34abc9dc5019e9e8b8; no voters:
I20260620 22:52:21.726233 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b0479133a4f4ba4d1e1c6c5c9ca07" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a04f613507e4819be7aa4faaab6681b" is_pre_election: true
I20260620 22:52:21.726323 9772 raft_consensus.cc:2781] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:21.726362 9772 raft_consensus.cc:482] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:21.726369 9557 raft_consensus.cc:2454] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 0.
I20260620 22:52:21.726392 9772 raft_consensus.cc:3037] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:21.727131 9772 raft_consensus.cc:504] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.727234 9772 leader_election.cc:288] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 election: Requested vote from peers f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525), 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:21.727375 9693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b0479133a4f4ba4d1e1c6c5c9ca07" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98740a4e4084f34abc9dc5019e9e8b8"
I20260620 22:52:21.727414 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b0479133a4f4ba4d1e1c6c5c9ca07" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a04f613507e4819be7aa4faaab6681b"
I20260620 22:52:21.727450 9693 raft_consensus.cc:3037] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:21.727481 9557 raft_consensus.cc:3037] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:21.728089 9693 raft_consensus.cc:2454] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 1.
I20260620 22:52:21.728186 9557 raft_consensus.cc:2454] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 1.
I20260620 22:52:21.728257 9354 leader_election.cc:302] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72a62afdf3f4af5a2adb6599aaa701f, f98740a4e4084f34abc9dc5019e9e8b8; no voters:
I20260620 22:52:21.728339 9772 raft_consensus.cc:2781] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:21.728484 9772 raft_consensus.cc:686] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 1 LEADER]: Becoming Leader. State: Replica: a72a62afdf3f4af5a2adb6599aaa701f, State: Running, Role: LEADER
I20260620 22:52:21.728570 9772 consensus_queue.cc:237] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } }
I20260620 22:52:21.729346 8503 catalog_manager.cc:5495] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f reported cstate change: term changed from 0 to 1, leader changed from <none> to a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129). New cstate: current_term: 1 leader_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } health_report { overall_health: HEALTHY } } }
I20260620 22:52:21.740744 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.738859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.2.90.129:37037)
0620 22:52:21.738897 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.738899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.738905 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:52:21.738957 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.738958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.738974 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.739026 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:21.739103 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.739541 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.740167 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.740348 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.740390 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.740490 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:21.740578 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:21.740629 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.740644 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.740657 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:21.742333 9693 raft_consensus.cc:1261] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Refusing update from remote peer a72a62afdf3f4af5a2adb6599aaa701f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:21.742333 9557 raft_consensus.cc:1261] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Refusing update from remote peer a72a62afdf3f4af5a2adb6599aaa701f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:21.742654 9772 consensus_queue.cc:1035] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:21.742781 9777 consensus_queue.cc:1035] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:21.745064 9780 mvcc.cc:205] Tried to move back new op lower bound from 7299055377374031872 to 7299055377320865792. Current Snapshot: MvccSnapshot[applied={T|T < 7299055377374031872}]
I20260620 22:52:21.748179 8502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:21.748271 8502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:21.749580 8502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb437d662bf4e8c90bc2a02c1b4af86: Sending DeleteTablet for 3 replicas of tablet 236b0479133a4f4ba4d1e1c6c5c9ca07
I20260620 22:52:21.749883 9401 tablet_service.cc:1514] Processing DeleteTablet for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:21 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:53576
I20260620 22:52:21.749897 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:21 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:55742
I20260620 22:52:21.749951 9673 tablet_service.cc:1514] Processing DeleteTablet for tablet 236b0479133a4f4ba4d1e1c6c5c9ca07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:21 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:49016
I20260620 22:52:21.750079 9791 tablet_replica.cc:330] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: stopping tablet replica
I20260620 22:52:21.750167 9790 tablet_replica.cc:330] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: stopping tablet replica
I20260620 22:52:21.750170 9791 raft_consensus.cc:2229] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:21.750242 9790 raft_consensus.cc:2229] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:21.750265 9791 raft_consensus.cc:2258] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:21.750332 9790 raft_consensus.cc:2258] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:21.750682 9791 ts_tablet_manager.cc:1905] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:21.750682 9790 ts_tablet_manager.cc:1905] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:21.751003 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8471
I20260620 22:52:21.752665 9792 tablet_replica.cc:330] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: stopping tablet replica
I20260620 22:52:21.752792 9792 raft_consensus.cc:2229] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:21.752813 9791 ts_tablet_manager.cc:1918] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:21.752894 9792 raft_consensus.cc:2258] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:21.752978 9790 ts_tablet_manager.cc:1918] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:21.753023 9791 log.cc:1196] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/236b0479133a4f4ba4d1e1c6c5c9ca07
I20260620 22:52:21.753042 9790 log.cc:1196] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/236b0479133a4f4ba4d1e1c6c5c9ca07
I20260620 22:52:21.753270 9790 ts_tablet_manager.cc:1939] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P a72a62afdf3f4af5a2adb6599aaa701f: Deleting consensus metadata
I20260620 22:52:21.753293 9791 ts_tablet_manager.cc:1939] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P 9a04f613507e4819be7aa4faaab6681b: Deleting consensus metadata
W20260620 22:52:21.755573 9493 connection.cc:535] server connection from 127.0.0.1:55742 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:52:21.755623 9357 connection.cc:535] server connection from 127.0.0.1:53576 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:52:21.756067 9792 ts_tablet_manager.cc:1905] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:21.757351 9792 ts_tablet_manager.cc:1918] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:21.757428 9792 log.cc:1196] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/236b0479133a4f4ba4d1e1c6c5c9ca07
I20260620 22:52:21.757861 9792 ts_tablet_manager.cc:1939] T 236b0479133a4f4ba4d1e1c6c5c9ca07 P f98740a4e4084f34abc9dc5019e9e8b8: Deleting consensus metadata
W20260620 22:52:21.758159 9629 connection.cc:441] server connection from 127.0.0.1:49016 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49016 (request call id 1) could send its response
I20260620 22:52:21.763643 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.757666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.2.90.188:39207)
0620 22:52:21.757705 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.757706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.757842 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:52:21.757994 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.757995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.758007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.758033 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.758664 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.759122 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.759786 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.759992 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.760046 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.761330 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.761331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.761331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.761337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.762617 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.762952 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.762954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.762954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.763011 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.763171 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.763172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.763173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.763283 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.763368 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.763470 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.763551 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:52:21.764428 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:21.757448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.2.90.189:34037)
0620 22:52:21.757490 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:21.757492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:21.757500 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:21.757520 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:21.757522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:21.757532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:21.757588 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:21.758266 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.759020 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.760065 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:21.760365 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:21.760405 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:21.761774 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:21.761775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:21.761775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:21.761782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:21.763372 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.763750 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.763751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.763752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.763808 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:21.763978 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:21.763979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:21.763979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:21.764103 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:21.764175 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:21.764300 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 22:52:21.764362 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:22.746353 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:22.746082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:55323)
0620 22:52:22.746215 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:22.746297 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
W20260620 22:52:22.746750 9740 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:22.747138 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:22.746869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:58161)
0620 22:52:22.747024 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:22.747094 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":46,"threads_started":1}
I20260620 22:52:22.747310 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:22.746586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:44735)
0620 22:52:22.747204 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:22.747269 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":50,"threads_started":1}
W20260620 22:52:22.747442 9604 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
W20260620 22:52:22.747499 9468 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35243 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:23.249590 9802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6fb437d662bf4e8c90bc2a02c1b4af86)
I20260620 22:52:23.249712 9802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } }
I20260620 22:52:23.249933 9802 leader_election.cc:288] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243), 5928a3b908dd40909a3a1882e237b0ec (127.2.90.189:34037)
I20260620 22:52:23.250224 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36384)
0620 22:52:23.250132 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.250189 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":42,"threads_started":1}
I20260620 22:52:23.250321 9806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6fb437d662bf4e8c90bc2a02c1b4af86)
W20260620 22:52:23.250384 9272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:23.250437 9806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } }
I20260620 22:52:23.250674 9806 leader_election.cc:288] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243), 3a53343101d145abad9bc883aa2f347c (127.2.90.188:39207)
I20260620 22:52:23.250780 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36400)
0620 22:52:23.250723 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.250748 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:52:23.250871 9272 leader_election.cc:334] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243): Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:23.250972 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36412)
0620 22:52:23.250866 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.250913 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":40,"threads_started":1}
W20260620 22:52:23.251053 9086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:23.251511 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.251310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36428)
0620 22:52:23.251470 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.251493 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260620 22:52:23.252203 9086 leader_election.cc:334] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243): Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:23.253479 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.2.90.189:34037)
0620 22:52:23.250301 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:23.250303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:23.250310 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:23.250334 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:23.250335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:23.250345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:23.250564 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:23.250597 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.251388 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.252700 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.253209 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.253279 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.253290 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:23.253406 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 22:52:23.253439 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":47,"threads_started":1}
I20260620 22:52:23.253544 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:34037 (local address 127.0.0.1:52920)
0620 22:52:23.250140 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.250150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:23.250205 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:23.250442 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:23.250443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:23.250445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:23.250505 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.250507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.251453 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.251454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.252609 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.252611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.253278 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.253278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.253333 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.253338 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:52:23.253391 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 22:52:23.253471 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":36,"thread_start_us":32,"threads_started":1}
I20260620 22:52:23.253739 9110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a53343101d145abad9bc883aa2f347c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5928a3b908dd40909a3a1882e237b0ec" is_pre_election: true
I20260620 22:52:23.253818 9110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a53343101d145abad9bc883aa2f347c in term 1.
I20260620 22:52:23.253996 9270 leader_election.cc:302] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a53343101d145abad9bc883aa2f347c, 5928a3b908dd40909a3a1882e237b0ec; no voters: 6fb437d662bf4e8c90bc2a02c1b4af86
I20260620 22:52:23.254112 9802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 22:52:23.254153 9802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6fb437d662bf4e8c90bc2a02c1b4af86)
I20260620 22:52:23.254177 9802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:52:23.254817 9802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } }
I20260620 22:52:23.254920 9802 leader_election.cc:288] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [CANDIDATE]: Term 2 election: Requested vote from peers 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243), 5928a3b908dd40909a3a1882e237b0ec (127.2.90.189:34037)
I20260620 22:52:23.255106 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:39207 (local address 127.0.0.1:37390)
0620 22:52:23.250969 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.250978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:23.251027 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:23.251236 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:23.251237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:23.251239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:23.251283 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.251284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.252154 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.252155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.253289 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.253291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.253987 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.253988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.254048 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.254052 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:23.255004 (+ 952us) client_negotiation.cc:240] Negotiation successful
0620 22:52:23.255047 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":59,"threads_started":1}
I20260620 22:52:23.255138 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.255014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36438)
0620 22:52:23.255061 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.255089 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:52:23.255146 9110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a53343101d145abad9bc883aa2f347c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5928a3b908dd40909a3a1882e237b0ec"
I20260620 22:52:23.255208 9110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:52:23.255141 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.250945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.2.90.188:39207)
0620 22:52:23.251112 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:23.251114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:23.251119 (+ 5us) server_negotiation.cc:407] Connection header received
0620 22:52:23.251137 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:23.251138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:23.251146 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:23.251223 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:23.251360 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.252082 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.253351 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.253916 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.253983 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.253993 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:23.255011 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0620 22:52:23.255055 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260620 22:52:23.255376 9295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5928a3b908dd40909a3a1882e237b0ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a53343101d145abad9bc883aa2f347c" is_pre_election: true
I20260620 22:52:23.255486 9295 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5928a3b908dd40909a3a1882e237b0ec in current term 2: Already voted for candidate 3a53343101d145abad9bc883aa2f347c in this term.
I20260620 22:52:23.255637 9087 leader_election.cc:302] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5928a3b908dd40909a3a1882e237b0ec; no voters: 3a53343101d145abad9bc883aa2f347c, 6fb437d662bf4e8c90bc2a02c1b4af86
I20260620 22:52:23.255949 9110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a53343101d145abad9bc883aa2f347c in term 2.
I20260620 22:52:23.256013 9806 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:52:23.256111 9270 leader_election.cc:302] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a53343101d145abad9bc883aa2f347c, 5928a3b908dd40909a3a1882e237b0ec; no voters:
I20260620 22:52:23.256218 9802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:52:23.256256 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.256137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36444)
0620 22:52:23.256181 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.256212 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:52:23.256306 9272 leader_election.cc:334] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243): Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
I20260620 22:52:23.256348 9802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [term 2 LEADER]: Becoming Leader. State: Replica: 3a53343101d145abad9bc883aa2f347c, State: Running, Role: LEADER
I20260620 22:52:23.256423 9802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } }
I20260620 22:52:23.256735 9812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a53343101d145abad9bc883aa2f347c. Latest consensus state: current_term: 2 leader_uuid: "3a53343101d145abad9bc883aa2f347c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:23.256821 9812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:23.256981 9814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:23.257340 9814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:23.257390 9814 catalog_manager.cc:1259] Loaded cluster ID: 204e316b818946a6934eb4439f9aff02
I20260620 22:52:23.257411 9814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:23.257529 9814 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:23.257577 9814 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Loaded TSK: 0
I20260620 22:52:23.257665 9814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:23.294266 9285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:23.295841 9285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:23.296829 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.296714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36464)
0620 22:52:23.296761 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.296798 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:52:23.296849 9110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [term 2 FOLLOWER]: Refusing update from remote peer 3a53343101d145abad9bc883aa2f347c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 22:52:23.297142 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.297048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.0.0.1:36474)
0620 22:52:23.297095 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.297121 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:52:23.297102 9802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 22:52:23.297221 9272 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c -> Peer 6fb437d662bf4e8c90bc2a02c1b4af86 (127.2.90.190:35243): Couldn't send request to peer 6fb437d662bf4e8c90bc2a02c1b4af86. Status: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 22:52:23.297703 9806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a53343101d145abad9bc883aa2f347c. Latest consensus state: current_term: 2 leader_uuid: "3a53343101d145abad9bc883aa2f347c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:23.297796 9806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:52:23.298156 9802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3a53343101d145abad9bc883aa2f347c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:23.298243 9802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:23.298311 9815 mvcc.cc:205] Tried to move back new op lower bound from 7299055383741943808 to 7299055383578873856. Current Snapshot: MvccSnapshot[applied={T|T < 7299055383741943808 or (T in {7299055383741943808})}]
I20260620 22:52:23.298377 9806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3a53343101d145abad9bc883aa2f347c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fb437d662bf4e8c90bc2a02c1b4af86" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35243 } } peers { permanent_uuid: "5928a3b908dd40909a3a1882e237b0ec" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 34037 } attrs { promote: false } } peers { permanent_uuid: "3a53343101d145abad9bc883aa2f347c" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 39207 } attrs { promote: false } } }
I20260620 22:52:23.298452 9806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5928a3b908dd40909a3a1882e237b0ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:52:23.302843 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.299881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37037 (local address 127.0.0.1:53590)
0620 22:52:23.300007 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.300015 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:23.300047 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:23.300252 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:23.300252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:23.300255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:23.300285 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.300286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.301272 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.301273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.302003 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.302005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.302629 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.302630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.302690 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.302694 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:23.302746 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:52:23.302779 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 22:52:23.302866 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.299970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.2.90.129:37037)
0620 22:52:23.300119 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:23.300120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:23.300127 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:23.300149 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:23.300150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:23.300160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:23.300389 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:23.300427 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.301184 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.302080 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.302567 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.302631 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.302642 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:23.302761 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:52:23.302796 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260620 22:52:23.303220 9401 tablet_service.cc:1467] Processing CreateTablet for tablet f6b463bc21264c65915b42b53597106c (DEFAULT_TABLE table=test-table [id=9677a7479d0e4d738ff3ef781a1372c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:23.303296 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.300074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.2.90.130:35623)
0620 22:52:23.300303 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:23.300305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:23.300312 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:23.300334 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:23.300335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:23.300343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:23.300395 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:23.300564 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.301284 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.302459 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.302991 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.303051 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.303063 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:23.303186 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 22:52:23.303225 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":48,"threads_started":1}
I20260620 22:52:23.303283 9803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.299847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:36525 (local address 127.0.0.1:49028)
0620 22:52:23.299891 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.299901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:23.299950 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:23.300146 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:23.300147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:23.300149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:23.300196 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.300198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.301015 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.301016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.302162 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.302165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.303120 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.303121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.303168 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.303172 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:23.303198 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 22:52:23.303226 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:52:23.303287 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.299923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35623 (local address 127.0.0.1:55754)
0620 22:52:23.300191 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.300200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:23.300255 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:23.300448 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:23.300449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:23.300450 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:23.300499 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.300500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.301358 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.301359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.302368 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:23.302370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.303052 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.303053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:23.303124 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.303128 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:23.303182 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:52:23.303219 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":61,"threads_started":1}
I20260620 22:52:23.303352 9401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b463bc21264c65915b42b53597106c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:23.303653 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.299897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.2.90.131:36525)
0620 22:52:23.300017 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:23.300019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:23.300027 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:23.300052 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:23.300053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:23.300064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:23.300129 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:23.300270 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.300948 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.302256 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.302815 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.303540 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.303553 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:23.303568 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 22:52:23.303602 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260620 22:52:23.304498 9825 tablet_bootstrap.cc:492] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: Bootstrap starting.
I20260620 22:52:23.304760 9673 tablet_service.cc:1467] Processing CreateTablet for tablet f6b463bc21264c65915b42b53597106c (DEFAULT_TABLE table=test-table [id=9677a7479d0e4d738ff3ef781a1372c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:23.304780 9537 tablet_service.cc:1467] Processing CreateTablet for tablet f6b463bc21264c65915b42b53597106c (DEFAULT_TABLE table=test-table [id=9677a7479d0e4d738ff3ef781a1372c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:23.304877 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b463bc21264c65915b42b53597106c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:23.304881 9673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b463bc21264c65915b42b53597106c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:23.304872 9825 tablet_bootstrap.cc:654] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:23.305444 9825 tablet_bootstrap.cc:492] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: No bootstrap required, opened a new log
I20260620 22:52:23.305485 9825 ts_tablet_manager.cc:1397] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:23.305598 9825 raft_consensus.cc:348] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.305657 9825 raft_consensus.cc:374] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:23.305676 9825 raft_consensus.cc:729] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72a62afdf3f4af5a2adb6599aaa701f, State: Initialized, Role: FOLLOWER
I20260620 22:52:23.305713 9825 consensus_queue.cc:260] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.305833 9825 ts_tablet_manager.cc:1428] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:23.306123 9828 tablet_bootstrap.cc:492] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Bootstrap starting.
I20260620 22:52:23.306265 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.306052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:38699)
0620 22:52:23.306200 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.306234 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260620 22:52:23.306463 9828 tablet_bootstrap.cc:654] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:23.306535 9830 tablet_bootstrap.cc:492] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Bootstrap starting.
I20260620 22:52:23.306581 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.306442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:58923)
0620 22:52:23.306538 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.306561 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260620 22:52:23.306861 9830 tablet_bootstrap.cc:654] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:23.307169 9828 tablet_bootstrap.cc:492] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: No bootstrap required, opened a new log
I20260620 22:52:23.307212 9828 ts_tablet_manager.cc:1397] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:23.307317 9828 raft_consensus.cc:348] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.307370 9830 tablet_bootstrap.cc:492] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: No bootstrap required, opened a new log
I20260620 22:52:23.307372 9828 raft_consensus.cc:374] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:23.307404 9830 ts_tablet_manager.cc:1397] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:23.307423 9828 raft_consensus.cc:729] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98740a4e4084f34abc9dc5019e9e8b8, State: Initialized, Role: FOLLOWER
I20260620 22:52:23.307462 9828 consensus_queue.cc:260] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.307526 9830 raft_consensus.cc:348] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.307587 9830 raft_consensus.cc:374] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:23.307590 9828 ts_tablet_manager.cc:1428] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:23.307614 9830 raft_consensus.cc:729] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a04f613507e4819be7aa4faaab6681b, State: Initialized, Role: FOLLOWER
I20260620 22:52:23.307677 9830 consensus_queue.cc:260] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.307814 9830 ts_tablet_manager.cc:1428] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:23.307997 9739 heartbeater.cc:503] Master 127.2.90.188:39207 was elected leader, sending a full tablet report...
I20260620 22:52:23.308189 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.307830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:34295)
0620 22:52:23.307955 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.307986 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260620 22:52:23.308436 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.308141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:45515)
0620 22:52:23.308354 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.308395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260620 22:52:23.308707 9603 heartbeater.cc:503] Master 127.2.90.188:39207 was elected leader, sending a full tablet report...
I20260620 22:52:23.308751 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.308453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:56793)
0620 22:52:23.308700 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.308729 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260620 22:52:23.308878 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.308803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:33939)
0620 22:52:23.308833 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.308856 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:52:23.412400 9827 raft_consensus.cc:482] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:23.412506 9827 raft_consensus.cc:504] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.412665 9827 leader_election.cc:288] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525), 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:23.412926 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b463bc21264c65915b42b53597106c" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a04f613507e4819be7aa4faaab6681b" is_pre_election: true
I20260620 22:52:23.412927 9693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b463bc21264c65915b42b53597106c" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" is_pre_election: true
I20260620 22:52:23.413024 9557 raft_consensus.cc:2454] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 0.
I20260620 22:52:23.413024 9693 raft_consensus.cc:2454] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 0.
I20260620 22:52:23.413174 9354 leader_election.cc:302] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a04f613507e4819be7aa4faaab6681b, a72a62afdf3f4af5a2adb6599aaa701f; no voters:
I20260620 22:52:23.413283 9827 raft_consensus.cc:2781] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:23.413326 9827 raft_consensus.cc:482] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:23.413349 9827 raft_consensus.cc:3037] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:23.414095 9827 raft_consensus.cc:504] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.414202 9827 leader_election.cc:288] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 election: Requested vote from peers f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525), 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623)
I20260620 22:52:23.414381 9693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b463bc21264c65915b42b53597106c" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98740a4e4084f34abc9dc5019e9e8b8"
I20260620 22:52:23.414454 9693 raft_consensus.cc:3037] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:23.414403 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b463bc21264c65915b42b53597106c" candidate_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a04f613507e4819be7aa4faaab6681b"
I20260620 22:52:23.414510 9557 raft_consensus.cc:3037] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:23.415004 9693 raft_consensus.cc:2454] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 1.
I20260620 22:52:23.415164 9354 leader_election.cc:302] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72a62afdf3f4af5a2adb6599aaa701f, f98740a4e4084f34abc9dc5019e9e8b8; no voters:
I20260620 22:52:23.415195 9557 raft_consensus.cc:2454] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72a62afdf3f4af5a2adb6599aaa701f in term 1.
I20260620 22:52:23.415256 9827 raft_consensus.cc:2781] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:23.415396 9827 raft_consensus.cc:686] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 1 LEADER]: Becoming Leader. State: Replica: a72a62afdf3f4af5a2adb6599aaa701f, State: Running, Role: LEADER
I20260620 22:52:23.415462 9827 consensus_queue.cc:237] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } }
I20260620 22:52:23.415493 9467 heartbeater.cc:503] Master 127.2.90.188:39207 was elected leader, sending a full tablet report...
I20260620 22:52:23.415746 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.415632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:60435)
0620 22:52:23.415679 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.415714 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:52:23.416026 9285 catalog_manager.cc:5495] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f reported cstate change: term changed from 0 to 1, leader changed from <none> to a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129). New cstate: current_term: 1 leader_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72a62afdf3f4af5a2adb6599aaa701f" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 } health_report { overall_health: UNKNOWN } } }
I20260620 22:52:23.445945 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.443948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.2.90.129:37037)
0620 22:52:23.444005 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:23.444006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:23.444017 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:23.444039 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:23.444039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:23.444051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:23.444096 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:23.444209 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.444657 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.445371 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:23.445551 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:23.445595 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:23.445688 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:23.445711 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:23.445750 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:23.445845 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 22:52:23.445864 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:52:23.446745 9557 raft_consensus.cc:1261] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Refusing update from remote peer a72a62afdf3f4af5a2adb6599aaa701f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:23.446774 9693 raft_consensus.cc:1261] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Refusing update from remote peer a72a62afdf3f4af5a2adb6599aaa701f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:23.446981 9827 consensus_queue.cc:1035] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98740a4e4084f34abc9dc5019e9e8b8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 36525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:23.447078 9838 consensus_queue.cc:1035] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a04f613507e4819be7aa4faaab6681b" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:23.447893 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.447794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:38049)
0620 22:52:23.447834 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.447861 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 22:52:23.447960 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.447826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:50043)
0620 22:52:23.447898 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.447932 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 22:52:23.448846 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.448756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:58811)
0620 22:52:23.448789 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.448820 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 22:52:23.449123 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.448947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.130:47781)
0620 22:52:23.449075 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.449100 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 22:52:23.449532 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.449420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.129:33415)
0620 22:52:23.449479 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.449508 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 22:52:23.449645 9842 mvcc.cc:205] Tried to move back new op lower bound from 7299055384356356096 to 7299055384230174720. Current Snapshot: MvccSnapshot[applied={T|T < 7299055384356356096}]
I20260620 22:52:23.449906 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:23.449826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35243 (local address 127.2.90.131:45813)
0620 22:52:23.449860 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.449882 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 22:52:23.450193 9285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:23.450235 9843 mvcc.cc:205] Tried to move back new op lower bound from 7299055384356356096 to 7299055384230174720. Current Snapshot: MvccSnapshot[applied={T|T < 7299055384356356096 or (T in {7299055384356356096})}]
I20260620 22:52:23.450278 9285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:23.451327 9285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a53343101d145abad9bc883aa2f347c: Sending DeleteTablet for 3 replicas of tablet f6b463bc21264c65915b42b53597106c
I20260620 22:52:23.451548 9673 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b463bc21264c65915b42b53597106c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:23 UTC) from {username='oryx', principal='oryx/127.2.90.188@KRBTEST.COM'} at 127.0.0.1:49028
I20260620 22:52:23.451610 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b463bc21264c65915b42b53597106c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:23 UTC) from {username='oryx', principal='oryx/127.2.90.188@KRBTEST.COM'} at 127.0.0.1:55754
I20260620 22:52:23.451625 9401 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b463bc21264c65915b42b53597106c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:23 UTC) from {username='oryx', principal='oryx/127.2.90.188@KRBTEST.COM'} at 127.0.0.1:53590
I20260620 22:52:23.451716 9850 tablet_replica.cc:330] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: stopping tablet replica
I20260620 22:52:23.451762 9850 raft_consensus.cc:2229] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:23.451769 9851 tablet_replica.cc:330] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: stopping tablet replica
I20260620 22:52:23.451795 9850 raft_consensus.cc:2258] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:23.451807 9851 raft_consensus.cc:2229] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:23.451843 9851 raft_consensus.cc:2258] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:23.451920 9852 tablet_replica.cc:330] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: stopping tablet replica
I20260620 22:52:23.451961 9852 raft_consensus.cc:2229] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:23.451977 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 9338
I20260620 22:52:23.452108 9851 ts_tablet_manager.cc:1905] T f6b463bc21264c65915b42b53597106c P a72a62afdf3f4af5a2adb6599aaa701f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:23.452116 9850 ts_tablet_manager.cc:1905] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:23.452026 9852 raft_consensus.cc:2258] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:23.452783 9852 ts_tablet_manager.cc:1905] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:23.453747 9850 ts_tablet_manager.cc:1918] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:23.453787 9850 log.cc:1196] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/f6b463bc21264c65915b42b53597106c
I20260620 22:52:23.454005 9850 ts_tablet_manager.cc:1939] T f6b463bc21264c65915b42b53597106c P f98740a4e4084f34abc9dc5019e9e8b8: Deleting consensus metadata
I20260620 22:52:23.454207 9852 ts_tablet_manager.cc:1918] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:23.454250 9852 log.cc:1196] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/f6b463bc21264c65915b42b53597106c
I20260620 22:52:23.454262 9270 catalog_manager.cc:4841] TS f98740a4e4084f34abc9dc5019e9e8b8 (127.2.90.131:36525): tablet f6b463bc21264c65915b42b53597106c (table test-table [id=9677a7479d0e4d738ff3ef781a1372c9]) successfully deleted
I20260620 22:52:23.454447 9852 ts_tablet_manager.cc:1939] T f6b463bc21264c65915b42b53597106c P 9a04f613507e4819be7aa4faaab6681b: Deleting consensus metadata
I20260620 22:52:23.454720 9270 catalog_manager.cc:4841] TS 9a04f613507e4819be7aa4faaab6681b (127.2.90.130:35623): tablet f6b463bc21264c65915b42b53597106c (table test-table [id=9677a7479d0e4d738ff3ef781a1372c9]) successfully deleted
W20260620 22:52:23.458506 9822 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:23.458385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37037 (local address 127.0.0.1:53610)
0620 22:52:23.458432 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:23.458460 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37037: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:52:23.458663 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 9474
W20260620 22:52:23.458654 9271 catalog_manager.cc:4584] TS a72a62afdf3f4af5a2adb6599aaa701f (127.2.90.129:37037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6b463bc21264c65915b42b53597106c: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37037: connect: Connection reset by peer (error 104)
I20260620 22:52:23.463174 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 9610
I20260620 22:52:23.467185 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 8967
I20260620 22:52:23.471086 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 9154
2026-06-20T22:52:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4519 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:23 dist-test-slave-4mtw krb5kdc[9855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:23 dist-test-slave-4mtw krb5kdc[9855](info): set up 2 sockets
Jun 20 22:52:23 dist-test-slave-4mtw krb5kdc[9855](info): commencing operation
krb5kdc: starting...
W20260620 22:52:24.420811 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab.
Entry for principal oryx/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab.
I20260620 22:52:24.526842 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39549
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab
--principal=oryx/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:24.604391 9868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:24.604589 9868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:24.604609 9868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:24.605772 9868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:24.605813 9868 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 22:52:24.605825 9868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:24.605837 9868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:24.605861 9868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:24.605883 9868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:24.607368 9868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:39549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.190.keytab
--principal=oryx/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:24.607591 9868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:24.607815 9868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:24.612272 9876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:24.612283 9873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:24.612396 9868 server_base.cc:1034] running on GCE node
W20260620 22:52:24.612251 9874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:24.612742 9868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:24.613032 9868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:24.614162 9868 hybrid_clock.cc:648] HybridClock initialized: now 1781995944614159 us; error 38 us; skew 500 ppm
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:24.616427 9868 init.cc:377] Logged in from keytab as oryx/127.2.90.190@KRBTEST.COM (short username oryx)
I20260620 22:52:24.616885 9868 webserver.cc:460] Webserver started at http://127.2.90.190:38123/ using document root <none> and password file <none>
I20260620 22:52:24.617097 9868 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:24.617141 9868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:24.617260 9868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:24.618286 9868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "88f26ade53224579912758e8c5773c0d"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "ab43fe7f258f5bf9e4d38c759e221869"
server_key_iv: "c802e64314a4d43bbc553af0d4297e7f"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.618616 9868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "88f26ade53224579912758e8c5773c0d"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "ab43fe7f258f5bf9e4d38c759e221869"
server_key_iv: "c802e64314a4d43bbc553af0d4297e7f"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.619797 9868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:24.620436 9883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:24.620574 9868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:24.620652 9868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "88f26ade53224579912758e8c5773c0d"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "ab43fe7f258f5bf9e4d38c759e221869"
server_key_iv: "c802e64314a4d43bbc553af0d4297e7f"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.620734 9868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:24.642859 9868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:24.644661 9868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:24.644811 9868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:24.649088 9868 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:39549
I20260620 22:52:24.649144 9945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:39549 every 8 connection(s)
I20260620 22:52:24.649451 9868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:24.650036 9946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:24.650826 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 9868
I20260620 22:52:24.650911 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:24.651089 2410 external_mini_cluster.cc:1428] Setting key 8169d4550fa571d3cef9a65fb4083243
I20260620 22:52:24.653030 9946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Bootstrap starting.
I20260620 22:52:24.653693 9946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:24.654001 9946 log.cc:824] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:24.654649 9946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: No bootstrap required, opened a new log
I20260620 22:52:24.656716 9946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f26ade53224579912758e8c5773c0d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39549 } }
I20260620 22:52:24.656867 9946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:24.656912 9946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f26ade53224579912758e8c5773c0d, State: Initialized, Role: FOLLOWER
I20260620 22:52:24.657047 9946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f26ade53224579912758e8c5773c0d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39549 } }
I20260620 22:52:24.657124 9946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:24.657155 9946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:24.657203 9946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 0 FOLLOWER]: Advancing to term 1
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:24.657905 9946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f26ade53224579912758e8c5773c0d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39549 } }
I20260620 22:52:24.657999 9946 leader_election.cc:302] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f26ade53224579912758e8c5773c0d; no voters:
I20260620 22:52:24.658146 9946 leader_election.cc:288] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:24.658174 9951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:24.658277 9951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [term 1 LEADER]: Becoming Leader. State: Replica: 88f26ade53224579912758e8c5773c0d, State: Running, Role: LEADER
I20260620 22:52:24.658360 9951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f26ade53224579912758e8c5773c0d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39549 } }
I20260620 22:52:24.658387 9946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:24.658705 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f26ade53224579912758e8c5773c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f26ade53224579912758e8c5773c0d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39549 } } }
I20260620 22:52:24.658774 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:24.658875 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f26ade53224579912758e8c5773c0d. Latest consensus state: current_term: 1 leader_uuid: "88f26ade53224579912758e8c5773c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f26ade53224579912758e8c5773c0d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39549 } } }
I20260620 22:52:24.658929 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:24.659425 9956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:24.659899 9956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:24.661458 9956 catalog_manager.cc:1347] Generated new cluster ID: 48ae8c1c48fa42a28c37abe6d0511d37
I20260620 22:52:24.661500 9956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:24.662117 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.651559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.2.90.190:39549)
0620 22:52:24.651736 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:24.651745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:24.651777 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:52:24.652216 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:24.652245 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:24.652311 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:24.652469 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:24.653055 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.654329 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.655068 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.655708 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.655823 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.658139 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:24.658143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:24.658143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:24.658152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:24.660465 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.660933 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.660935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.660935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.661012 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.661233 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.661234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.661234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.661540 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:24.661679 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:24.661794 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 22:52:24.661928 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260620 22:52:24.677995 9956 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:24.678711 9956 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:24.689749 9956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Generated new TSK 0
I20260620 22:52:24.689922 9956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab.
Entry for principal oryx/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab.
I20260620 22:52:24.715296 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39549
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:24.792799 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:24.792980 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:24.793002 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:24.794041 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:24.794072 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:24.794090 9972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:52:24.794111 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:24.795465 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:24.795626 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:24.795789 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:24.800179 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:24.800225 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:24.800371 9972 server_base.cc:1034] running on GCE node
W20260620 22:52:24.800189 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:24.800621 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:24.800872 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:24.802012 9972 hybrid_clock.cc:648] HybridClock initialized: now 1781995944801999 us; error 28 us; skew 500 ppm
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:24.804280 9972 init.cc:377] Logged in from keytab as oryx/127.2.90.129@KRBTEST.COM (short username oryx)
I20260620 22:52:24.804628 9972 webserver.cc:460] Webserver started at http://127.2.90.129:44725/ using document root <none> and password file <none>
I20260620 22:52:24.804831 9972 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:24.804875 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:24.804960 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:24.805868 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "e612d19296534eb18d1ac3e715d3deb3"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "b6537805a487d8ba57572ff2657e90f8"
server_key_iv: "b3616aa8eec939967df4a8cc27e843ed"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.806136 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "e612d19296534eb18d1ac3e715d3deb3"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "b6537805a487d8ba57572ff2657e90f8"
server_key_iv: "b3616aa8eec939967df4a8cc27e843ed"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.807222 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:24.807850 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:24.808017 9972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:24.808085 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "e612d19296534eb18d1ac3e715d3deb3"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "b6537805a487d8ba57572ff2657e90f8"
server_key_iv: "b3616aa8eec939967df4a8cc27e843ed"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.808151 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:24.818948 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:24.820482 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:24.820593 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:24.821055 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:24.821099 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:24.821118 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:24.821132 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:24.827116 9972 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:36075
I20260620 22:52:24.827193 10102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:36075 every 8 connection(s)
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:24.827478 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:24.829303 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 9972
I20260620 22:52:24.829419 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:24.829607 2410 external_mini_cluster.cc:1428] Setting key 9c79522f8eadf2907d7d05d84f54bad2
I20260620 22:52:24.830161 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.821461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:45799 (local address 127.2.90.190:39549)
0620 22:52:24.821530 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:24.821532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:24.821671 (+ 139us) server_negotiation.cc:407] Connection header received
0620 22:52:24.822111 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:24.822112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:24.822123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:24.822199 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:24.822909 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.823671 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.825149 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.825452 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.825532 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.827787 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:24.827788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:24.827789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:24.827796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:24.828879 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.829217 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.829218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.829218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.829283 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.829514 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.829515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.829516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.829627 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:24.829683 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:24.829975 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 22:52:24.830071 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:52:24.830273 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.821399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.2.90.129:45799)
0620 22:52:24.821552 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:24.821589 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:24.822021 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:24.822248 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:24.822251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:24.822566 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:24.822819 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.822825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.823761 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.823762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.825063 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.825066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.825535 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.825536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.825626 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.826078 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:24.826088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:24.827692 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:24.828944 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.828950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.828952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.829148 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.829338 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.829338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.829339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.829464 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.829683 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:24.829689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:24.829843 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:52:24.829939 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:52:24.830082 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260620 22:52:24.830675 10103 heartbeater.cc:347] Connected to a master server at 127.2.90.190:39549
I20260620 22:52:24.830760 10103 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:24.830935 10103 heartbeater.cc:511] Master 127.2.90.190:39549 requested a full tablet report, sending...
I20260620 22:52:24.831467 9899 ts_manager.cc:194] Registered new tserver with Master: e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075)
I20260620 22:52:24.832371 9899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:45799
I20260620 22:52:24.836905 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.831577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:54733 (local address 127.2.90.190:39549)
0620 22:52:24.831609 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:24.831610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:24.831837 (+ 227us) server_negotiation.cc:407] Connection header received
0620 22:52:24.831886 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:24.831886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:24.831896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:24.831931 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:24.832203 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.832814 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.833703 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.833872 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.833921 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.834756 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:24.834757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:24.834757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:24.834762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:24.835923 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.836261 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.836261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.836261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.836303 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.836481 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.836481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.836481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.836568 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:24.836611 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:24.836796 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 22:52:24.836850 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:52:24.836959 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.831647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.2.90.129:54733)
0620 22:52:24.831770 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:24.831779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:24.831832 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:24.831947 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:24.831948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:24.832099 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:24.832141 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.832142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.832903 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.832904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.833625 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.833628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.833941 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.833942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.834015 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.834342 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:24.834347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:24.834678 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:24.835998 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.835999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.836000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.836193 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.836363 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.836363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.836364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.836427 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.836623 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:24.836623 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:24.836668 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 22:52:24.836778 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:52:24.836865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab.
Entry for principal oryx/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab.
I20260620 22:52:24.854626 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39549
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:24.935315 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:24.935520 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:24.935542 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:24.936636 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:24.936669 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:24.936717 10112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:52:24.936741 10112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:24.938191 10112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:24.938383 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:24.938551 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:24.943082 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:24.943133 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:24.943146 10112 server_base.cc:1034] running on GCE node
W20260620 22:52:24.943077 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:24.943442 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:24.943647 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:24.944749 10112 hybrid_clock.cc:648] HybridClock initialized: now 1781995944944730 us; error 33 us; skew 500 ppm
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:24.946915 10112 init.cc:377] Logged in from keytab as oryx/127.2.90.130@KRBTEST.COM (short username oryx)
I20260620 22:52:24.947353 10112 webserver.cc:460] Webserver started at http://127.2.90.130:46827/ using document root <none> and password file <none>
I20260620 22:52:24.947549 10112 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:24.947595 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:24.947722 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:24.948748 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "8864618b57b344cf882362bde1a7ab61"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "035e2e36b6514e6ac6a88faec831e350"
server_key_iv: "79924a0e2f7653dfa8918330b29fd0d2"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.949072 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "8864618b57b344cf882362bde1a7ab61"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "035e2e36b6514e6ac6a88faec831e350"
server_key_iv: "79924a0e2f7653dfa8918330b29fd0d2"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.950244 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:24.950876 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:24.951038 10112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:24.951119 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "8864618b57b344cf882362bde1a7ab61"
format_stamp: "Formatted at 2026-06-20 22:52:24 on dist-test-slave-4mtw"
server_key: "035e2e36b6514e6ac6a88faec831e350"
server_key_iv: "79924a0e2f7653dfa8918330b29fd0d2"
server_key_version: "encryptionkey@0"
I20260620 22:52:24.951184 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:24.959625 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:24.961573 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:24.961699 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:24.962186 10112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:24.962231 10112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:24.962256 10112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:24.962281 10112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:52:24 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:24.968789 10112 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:42993
I20260620 22:52:24.968885 10242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:42993 every 8 connection(s)
I20260620 22:52:24.969161 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:24.970814 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.962540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:48813 (local address 127.2.90.190:39549)
0620 22:52:24.962602 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:24.962604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:24.962777 (+ 173us) server_negotiation.cc:407] Connection header received
0620 22:52:24.963418 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:24.963427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:24.963437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:24.963483 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:24.964277 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.964947 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.965901 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.966091 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.966145 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.968478 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:24.968480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:24.968480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:24.968487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:24.969842 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.970155 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.970156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.970156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.970205 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.970369 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.970370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.970370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.970458 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:24.970499 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:24.970688 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:52:24.970740 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:52:24.970978 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.962518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.2.90.130:48813)
0620 22:52:24.962653 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:24.962696 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:24.963326 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:24.963573 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:24.963578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:24.963915 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:24.964198 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.964203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.965032 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.965033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.965818 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.965820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.966167 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.966168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.966278 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.966699 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:24.966707 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:24.968369 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:24.969905 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.969908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.969909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.970097 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.970253 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.970253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.970254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.970334 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.970498 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:24.970500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:24.970599 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 22:52:24.970676 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:52:24.970782 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260620 22:52:24.971382 10243 heartbeater.cc:347] Connected to a master server at 127.2.90.190:39549
I20260620 22:52:24.971458 10243 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:24.971607 10243 heartbeater.cc:511] Master 127.2.90.190:39549 requested a full tablet report, sending...
I20260620 22:52:24.971930 9900 ts_manager.cc:194] Registered new tserver with Master: 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:24.972431 9900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:48813
I20260620 22:52:24.977142 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.972215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.2.90.130:40361)
0620 22:52:24.972333 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:24.972344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:24.972398 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:24.972530 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:24.972530 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:24.972718 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:24.972774 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.972776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.973390 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.973391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.973900 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:24.973901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.974230 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.974231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:24.974285 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.974656 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:24.974664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:24.974911 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:24.976004 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.976005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.976005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.976153 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.976374 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:24.976375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:24.976375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:24.976440 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:24.976790 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:24.976791 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:24.976838 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 22:52:24.976941 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 22:52:24.977043 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260620 22:52:24.977145 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:24.972317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:40361 (local address 127.2.90.190:39549)
0620 22:52:24.972352 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:24.972353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:24.972386 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:52:24.972447 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:24.972448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:24.972455 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:24.972494 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:24.972843 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.973285 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.973973 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:24.974154 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:24.974202 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:24.974974 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:24.974975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:24.974975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:24.974979 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:24.975940 (+ 961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.976236 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.976237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.976237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.976290 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:24.976508 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:24.976509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:24.976509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:24.976720 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:24.976798 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:24.976981 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 22:52:24.977059 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:52:24.978673 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 10112
I20260620 22:52:24.978762 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:24.978951 2410 external_mini_cluster.cc:1428] Setting key 2974041c9c7b6440ec82a584e21bc97a
WARNING: no policy specified for oryx/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab.
Entry for principal oryx/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab.
I20260620 22:52:25.001618 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39549
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:25.079223 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:25.079428 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:25.079452 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:25.080562 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:25.080601 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:25.080621 10252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 22:52:25.080636 10252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:25.082099 10252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.90.190:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:25.082312 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:25.082525 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:25.087121 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:25.087121 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:25.087203 10252 server_base.cc:1034] running on GCE node
W20260620 22:52:25.087131 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:25.087551 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:25.087772 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:25.088899 10252 hybrid_clock.cc:648] HybridClock initialized: now 1781995945088888 us; error 28 us; skew 500 ppm
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995945, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:25.091136 10252 init.cc:377] Logged in from keytab as oryx/127.2.90.131@KRBTEST.COM (short username oryx)
I20260620 22:52:25.091560 10252 webserver.cc:460] Webserver started at http://127.2.90.131:36389/ using document root <none> and password file <none>
I20260620 22:52:25.091766 10252 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:25.091814 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:25.091934 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:25.092983 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "59cc15518a7443e3838631a8eaa249ca"
format_stamp: "Formatted at 2026-06-20 22:52:25 on dist-test-slave-4mtw"
server_key: "1fe9485ac9379d2cc6ebd93b2f16ed7c"
server_key_iv: "c0112d4ca599f683a9eb4f5172dc97f1"
server_key_version: "encryptionkey@0"
I20260620 22:52:25.093293 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "59cc15518a7443e3838631a8eaa249ca"
format_stamp: "Formatted at 2026-06-20 22:52:25 on dist-test-slave-4mtw"
server_key: "1fe9485ac9379d2cc6ebd93b2f16ed7c"
server_key_iv: "c0112d4ca599f683a9eb4f5172dc97f1"
server_key_version: "encryptionkey@0"
I20260620 22:52:25.094548 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:25.095142 10267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.095295 10252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:25.095377 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "59cc15518a7443e3838631a8eaa249ca"
format_stamp: "Formatted at 2026-06-20 22:52:25 on dist-test-slave-4mtw"
server_key: "1fe9485ac9379d2cc6ebd93b2f16ed7c"
server_key_iv: "c0112d4ca599f683a9eb4f5172dc97f1"
server_key_version: "encryptionkey@0"
I20260620 22:52:25.095441 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:25.105166 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:25.107007 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:25.107132 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:25.107667 10252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:25.107712 10252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.107741 10252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:25.107764 10252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995945, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:25.114276 10252 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:37825
I20260620 22:52:25.114372 10382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:37825 every 8 connection(s)
I20260620 22:52:25.114661 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:25.115770 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 10252
I20260620 22:52:25.115866 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:25.116066 2410 external_mini_cluster.cc:1428] Setting key 35c36270e31db706ecc1f311053cc756
I20260620 22:52:25.116803 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.108038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:41191 (local address 127.2.90.190:39549)
0620 22:52:25.108102 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.108105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.108253 (+ 148us) server_negotiation.cc:407] Connection header received
0620 22:52:25.108725 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.108726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.108739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.108812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.109442 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.109921 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.111430 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.111717 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.111790 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.114035 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:25.114037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:25.114037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:25.114044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:25.115484 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.115826 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.115827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.115827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.115888 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.116110 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.116111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.116112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.116232 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:25.116286 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.116622 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.116718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 22:52:25.116983 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.107995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.2.90.131:41191)
0620 22:52:25.108136 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.108173 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.108620 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.108889 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.108893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.109168 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:25.109344 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.109348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.110008 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.110009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.111333 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.111336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.111822 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.111823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.111975 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.112472 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:25.112480 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:25.113949 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:25.115552 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:25.115556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:25.115558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:25.115756 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:25.115945 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:25.115946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:25.115946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:25.116050 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:25.116351 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:25.116356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:25.116486 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.116592 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.116767 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260620 22:52:25.117388 10383 heartbeater.cc:347] Connected to a master server at 127.2.90.190:39549
I20260620 22:52:25.117457 10383 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:25.117602 10383 heartbeater.cc:511] Master 127.2.90.190:39549 requested a full tablet report, sending...
I20260620 22:52:25.117933 9899 ts_manager.cc:194] Registered new tserver with Master: 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825)
I20260620 22:52:25.118340 9899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:41191
I20260620 22:52:25.122764 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.118315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:60807 (local address 127.2.90.190:39549)
0620 22:52:25.118348 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.118350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.118443 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:52:25.118513 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.118514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.118524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.118569 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.118879 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.119327 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.119943 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.120106 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.120147 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.120878 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:25.120878 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:25.120878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:25.120884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:25.122018 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.122250 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.122251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.122251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.122290 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.122392 (+ 102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.122392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.122392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.122492 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:25.122537 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.122660 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.122712 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:52:25.122776 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.118266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.2.90.131:60807)
0620 22:52:25.118392 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.118401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.118456 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.118572 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.118573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.118741 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:25.118813 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.118815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.119404 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.119405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.119884 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.119885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.120153 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.120154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.120194 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.120519 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:25.120522 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:25.120816 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:25.122074 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:25.122074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:25.122075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:25.122204 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:25.122322 (+ 118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:25.122323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:25.122323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:25.122361 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:25.122537 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:25.122538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:25.122575 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.122644 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.122712 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260620 22:52:25.127626 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:52:25.146814 9949 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:25.142038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.2.90.190:39549)
0620 22:52:25.142089 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.142091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.142099 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:25.142137 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.142137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.142146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.142183 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.142672 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.143149 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.143824 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.144000 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.144042 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.146735 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33836: BlockingRecv error: recv got EOF from 127.0.0.1:33836 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:25.153203 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.147505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.2.90.190:39549)
0620 22:52:25.147567 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.147569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.147610 (+ 41us) server_negotiation.cc:407] Connection header received
0620 22:52:25.147647 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.147647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.147657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.147699 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.148060 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.148514 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.149149 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.149320 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.149364 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.151168 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:25.151170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:25.151170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:25.151176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:25.152379 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.152634 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.152634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.152634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.152696 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.152837 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.152838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.152838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.152933 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:25.153002 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.153075 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.153127 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:52:25.154547 9899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:25.156060 9899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:25.162271 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.157994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:37825 (local address 127.0.0.1:35562)
0620 22:52:25.158297 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.158368 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.158431 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.158694 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.158698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.158699 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.158761 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.158763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.160031 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.160032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.161038 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.161041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.161982 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.161983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.162059 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.162074 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.162135 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.162187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260620 22:52:25.162464 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.158007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42993 (local address 127.0.0.1:60824)
0620 22:52:25.158329 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.158368 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.158431 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.158699 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.158700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.158703 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.158770 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.158772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.159738 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.159739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.160786 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.160788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.162005 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.162006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.162083 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.162086 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.162130 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.162190 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260620 22:52:25.162635 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.158123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.2.90.130:42993)
0620 22:52:25.158417 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.158422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.158433 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:25.158523 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.158526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.158542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.158624 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.158841 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.159649 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.160931 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.161907 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.161961 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.162307 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.162361 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.162430 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":68,"threads_started":1}
I20260620 22:52:25.162688 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.158096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.2.90.131:37825)
0620 22:52:25.158253 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.158259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.158506 (+ 247us) server_negotiation.cc:407] Connection header received
0620 22:52:25.158548 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.158550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.158565 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.158628 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.158832 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.159952 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.161118 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.161796 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.161860 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.161949 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.162564 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.162603 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260620 22:52:25.163283 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.158215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36075 (local address 127.0.0.1:38722)
0620 22:52:25.158516 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.158523 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.158562 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.158844 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.158845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.158846 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.158876 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.158878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.160284 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.160285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.161316 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.161319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.162632 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.162633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.163105 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.163112 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.163168 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.163210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":3,"thread_start_us":35,"threads_started":1}
I20260620 22:52:25.163326 10317 tablet_service.cc:1467] Processing CreateTablet for tablet 3c63a2e3c8f44b989b271a1566afc03e (DEFAULT_TABLE table=test-table [id=d6bec0e223db47efa9b700ae3c1e1fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.163444 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 3c63a2e3c8f44b989b271a1566afc03e (DEFAULT_TABLE table=test-table [id=d6bec0e223db47efa9b700ae3c1e1fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.163709 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c63a2e3c8f44b989b271a1566afc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.163820 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c63a2e3c8f44b989b271a1566afc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.163852 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.158340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.2.90.129:36075)
0620 22:52:25.158521 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.158525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.158662 (+ 137us) server_negotiation.cc:407] Connection header received
0620 22:52:25.158703 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.158705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.158718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.158784 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.158958 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.160059 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.161406 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.162147 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.162216 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.162290 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.163663 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.163708 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260620 22:52:25.164605 10037 tablet_service.cc:1467] Processing CreateTablet for tablet 3c63a2e3c8f44b989b271a1566afc03e (DEFAULT_TABLE table=test-table [id=d6bec0e223db47efa9b700ae3c1e1fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.164958 10037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c63a2e3c8f44b989b271a1566afc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.168169 10409 tablet_bootstrap.cc:492] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Bootstrap starting.
I20260620 22:52:25.168637 10410 tablet_bootstrap.cc:492] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Bootstrap starting.
I20260620 22:52:25.168768 10409 tablet_bootstrap.cc:654] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.169085 10409 log.cc:824] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:25.169303 10410 tablet_bootstrap.cc:654] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.169536 10410 log.cc:824] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:25.169926 10409 tablet_bootstrap.cc:492] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: No bootstrap required, opened a new log
I20260620 22:52:25.169994 10409 ts_tablet_manager.cc:1397] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:52:25.170126 10408 tablet_bootstrap.cc:492] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Bootstrap starting.
I20260620 22:52:25.170203 10410 tablet_bootstrap.cc:492] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: No bootstrap required, opened a new log
I20260620 22:52:25.170262 10410 ts_tablet_manager.cc:1397] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:25.170639 10408 tablet_bootstrap.cc:654] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.170935 10408 log.cc:824] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:25.171540 10408 tablet_bootstrap.cc:492] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: No bootstrap required, opened a new log
I20260620 22:52:25.171594 10408 ts_tablet_manager.cc:1397] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:25.172024 10409 raft_consensus.cc:348] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.172106 10410 raft_consensus.cc:348] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.172248 10409 raft_consensus.cc:374] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.172298 10410 raft_consensus.cc:374] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.172297 10409 raft_consensus.cc:729] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.172340 10410 raft_consensus.cc:729] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e612d19296534eb18d1ac3e715d3deb3, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.172434 10410 consensus_queue.cc:260] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.172453 10409 consensus_queue.cc:260] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.172710 10410 ts_tablet_manager.cc:1428] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:52:25.172739 10383 heartbeater.cc:503] Master 127.2.90.190:39549 was elected leader, sending a full tablet report...
I20260620 22:52:25.172818 10103 heartbeater.cc:503] Master 127.2.90.190:39549 was elected leader, sending a full tablet report...
I20260620 22:52:25.172955 10409 ts_tablet_manager.cc:1428] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 22:52:25.173512 10408 raft_consensus.cc:348] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.173663 10408 raft_consensus.cc:374] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.173705 10408 raft_consensus.cc:729] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8864618b57b344cf882362bde1a7ab61, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.173812 10408 consensus_queue.cc:260] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.174054 10408 ts_tablet_manager.cc:1428] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:25.174096 10243 heartbeater.cc:503] Master 127.2.90.190:39549 was elected leader, sending a full tablet report...
W20260620 22:52:25.219749 10244 tablet.cc:2369] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:25.309270 10416 raft_consensus.cc:482] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:25.309383 10416 raft_consensus.cc:504] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.309671 10416 leader_election.cc:288] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825), e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075)
I20260620 22:52:25.312624 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.309961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:55887 (local address 127.2.90.129:36075)
0620 22:52:25.310002 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.310005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.310078 (+ 73us) server_negotiation.cc:407] Connection header received
0620 22:52:25.310121 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.310122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.310132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.310171 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.310293 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.310976 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.311940 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.312301 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.312355 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.312363 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.312516 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.312553 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:25.312630 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.309877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:51851 (local address 127.2.90.131:37825)
0620 22:52:25.309914 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.309915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.309932 (+ 17us) server_negotiation.cc:407] Connection header received
0620 22:52:25.310022 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.310024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.310034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.310082 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.310237 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.311010 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.311962 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.312323 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.312386 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.312394 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.312532 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.312567 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:25.312646 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.309828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:37825 (local address 127.2.90.130:51851)
0620 22:52:25.309879 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.309889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.309929 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.310087 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.310088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.310089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.310158 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.310160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.311093 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.311094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.311863 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.311866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.312396 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.312397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.312466 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.312469 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.312517 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.312554 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":17}
I20260620 22:52:25.312652 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.309843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36075 (local address 127.2.90.130:55887)
0620 22:52:25.309999 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.310025 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.310075 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.310174 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.310174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.310175 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.310235 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.310236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.311056 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.311056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.311850 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.311853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.312374 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.312375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.312451 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.312454 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.312492 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.312531 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260620 22:52:25.313032 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c63a2e3c8f44b989b271a1566afc03e" candidate_uuid: "8864618b57b344cf882362bde1a7ab61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3" is_pre_election: true
I20260620 22:52:25.313050 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c63a2e3c8f44b989b271a1566afc03e" candidate_uuid: "8864618b57b344cf882362bde1a7ab61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cc15518a7443e3838631a8eaa249ca" is_pre_election: true
I20260620 22:52:25.313184 10046 raft_consensus.cc:2454] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8864618b57b344cf882362bde1a7ab61 in term 0.
I20260620 22:52:25.313186 10337 raft_consensus.cc:2454] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8864618b57b344cf882362bde1a7ab61 in term 0.
I20260620 22:52:25.313380 10131 leader_election.cc:302] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, 8864618b57b344cf882362bde1a7ab61; no voters:
I20260620 22:52:25.313503 10416 raft_consensus.cc:2781] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:25.313555 10416 raft_consensus.cc:482] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:25.313583 10416 raft_consensus.cc:3037] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.314447 10416 raft_consensus.cc:504] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.314565 10416 leader_election.cc:288] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [CANDIDATE]: Term 1 election: Requested vote from peers 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825), e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075)
I20260620 22:52:25.314684 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c63a2e3c8f44b989b271a1566afc03e" candidate_uuid: "8864618b57b344cf882362bde1a7ab61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3"
I20260620 22:52:25.314754 10046 raft_consensus.cc:3037] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.314747 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c63a2e3c8f44b989b271a1566afc03e" candidate_uuid: "8864618b57b344cf882362bde1a7ab61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cc15518a7443e3838631a8eaa249ca"
I20260620 22:52:25.314817 10337 raft_consensus.cc:3037] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.315390 10046 raft_consensus.cc:2454] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8864618b57b344cf882362bde1a7ab61 in term 1.
I20260620 22:52:25.315538 10337 raft_consensus.cc:2454] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8864618b57b344cf882362bde1a7ab61 in term 1.
I20260620 22:52:25.315554 10128 leader_election.cc:302] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8864618b57b344cf882362bde1a7ab61, e612d19296534eb18d1ac3e715d3deb3; no voters:
I20260620 22:52:25.315722 10416 raft_consensus.cc:2781] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:25.315868 10416 raft_consensus.cc:686] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 1 LEADER]: Becoming Leader. State: Replica: 8864618b57b344cf882362bde1a7ab61, State: Running, Role: LEADER
I20260620 22:52:25.315953 10416 consensus_queue.cc:237] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } }
I20260620 22:52:25.316723 9899 catalog_manager.cc:5495] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8864618b57b344cf882362bde1a7ab61 (127.2.90.130). New cstate: current_term: 1 leader_uuid: "8864618b57b344cf882362bde1a7ab61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } health_report { overall_health: UNKNOWN } } }
W20260620 22:52:25.328115 10104 tablet.cc:2369] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:25.345275 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.343071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.2.90.130:42993)
0620 22:52:25.343122 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.343124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.343132 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:25.343201 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.343202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.343222 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.343280 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:25.343402 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.343846 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.344569 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.344785 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.344839 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.344986 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.345082 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.345148 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.345187 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.345206 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:52:25.346870 10046 raft_consensus.cc:1261] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Refusing update from remote peer 8864618b57b344cf882362bde1a7ab61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.346889 10337 raft_consensus.cc:1261] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Refusing update from remote peer 8864618b57b344cf882362bde1a7ab61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.347136 10416 consensus_queue.cc:1035] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.347273 10418 consensus_queue.cc:1035] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.351507 9900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.351588 9900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.352288 9900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Sending DeleteTablet for 3 replicas of tablet 3c63a2e3c8f44b989b271a1566afc03e
I20260620 22:52:25.352622 10317 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c63a2e3c8f44b989b271a1566afc03e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:35562
I20260620 22:52:25.352672 10037 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c63a2e3c8f44b989b271a1566afc03e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:38722
I20260620 22:52:25.352671 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c63a2e3c8f44b989b271a1566afc03e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:60824
I20260620 22:52:25.352840 10431 tablet_replica.cc:330] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: stopping tablet replica
I20260620 22:52:25.352895 10432 tablet_replica.cc:330] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: stopping tablet replica
I20260620 22:52:25.352932 10431 raft_consensus.cc:2229] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.352964 10432 raft_consensus.cc:2229] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:25.353022 10431 raft_consensus.cc:2258] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.353039 10432 raft_consensus.cc:2258] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.353253 10430 tablet_replica.cc:330] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: stopping tablet replica
I20260620 22:52:25.353266 9900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 22:52:25.353358 10430 raft_consensus.cc:2229] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 22:52:25.353404 9900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:25.353451 10430 raft_consensus.cc:2258] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.353751 10432 ts_tablet_manager.cc:1905] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.353889 10431 ts_tablet_manager.cc:1905] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.354976 10037 tablet_service.cc:1467] Processing CreateTablet for tablet 73e3cf443d874bb8ac53ae3da35a8d85 (DEFAULT_TABLE table=test-table [id=9c3abcfa476f475fb0cd3d9f971b3fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.355041 10430 ts_tablet_manager.cc:1905] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.355166 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 73e3cf443d874bb8ac53ae3da35a8d85 (DEFAULT_TABLE table=test-table [id=9c3abcfa476f475fb0cd3d9f971b3fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.355255 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e3cf443d874bb8ac53ae3da35a8d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.355568 10317 tablet_service.cc:1467] Processing CreateTablet for tablet 73e3cf443d874bb8ac53ae3da35a8d85 (DEFAULT_TABLE table=test-table [id=9c3abcfa476f475fb0cd3d9f971b3fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.355618 10432 ts_tablet_manager.cc:1918] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:25.355666 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e3cf443d874bb8ac53ae3da35a8d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.355685 10432 log.cc:1196] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/3c63a2e3c8f44b989b271a1566afc03e
I20260620 22:52:25.355903 10432 ts_tablet_manager.cc:1939] T 3c63a2e3c8f44b989b271a1566afc03e P 8864618b57b344cf882362bde1a7ab61: Deleting consensus metadata
I20260620 22:52:25.356770 10037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73e3cf443d874bb8ac53ae3da35a8d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.356964 10409 tablet_bootstrap.cc:492] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Bootstrap starting.
I20260620 22:52:25.357052 9884 catalog_manager.cc:4841] TS 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993): tablet 3c63a2e3c8f44b989b271a1566afc03e (table test-table [id=d6bec0e223db47efa9b700ae3c1e1fd6]) successfully deleted
I20260620 22:52:25.357347 10409 tablet_bootstrap.cc:654] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.357445 10431 ts_tablet_manager.cc:1918] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:25.357502 10431 log.cc:1196] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/3c63a2e3c8f44b989b271a1566afc03e
I20260620 22:52:25.357755 10431 ts_tablet_manager.cc:1939] T 3c63a2e3c8f44b989b271a1566afc03e P e612d19296534eb18d1ac3e715d3deb3: Deleting consensus metadata
I20260620 22:52:25.358094 9884 catalog_manager.cc:4841] TS e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075): tablet 3c63a2e3c8f44b989b271a1566afc03e (table test-table [id=d6bec0e223db47efa9b700ae3c1e1fd6]) successfully deleted
I20260620 22:52:25.358120 10430 ts_tablet_manager.cc:1918] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:25.358170 10430 log.cc:1196] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/3c63a2e3c8f44b989b271a1566afc03e
I20260620 22:52:25.358340 10430 ts_tablet_manager.cc:1939] T 3c63a2e3c8f44b989b271a1566afc03e P 59cc15518a7443e3838631a8eaa249ca: Deleting consensus metadata
I20260620 22:52:25.358610 10408 tablet_bootstrap.cc:492] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Bootstrap starting.
I20260620 22:52:25.358639 10410 tablet_bootstrap.cc:492] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Bootstrap starting.
I20260620 22:52:25.358650 9887 catalog_manager.cc:4841] TS 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825): tablet 3c63a2e3c8f44b989b271a1566afc03e (table test-table [id=d6bec0e223db47efa9b700ae3c1e1fd6]) successfully deleted
I20260620 22:52:25.358966 10408 tablet_bootstrap.cc:654] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.359022 10410 tablet_bootstrap.cc:654] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.359511 10408 tablet_bootstrap.cc:492] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: No bootstrap required, opened a new log
I20260620 22:52:25.359560 10408 ts_tablet_manager.cc:1397] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.359694 10408 raft_consensus.cc:348] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.359771 10408 raft_consensus.cc:374] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.359787 10408 raft_consensus.cc:729] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8864618b57b344cf882362bde1a7ab61, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.359820 10410 tablet_bootstrap.cc:492] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: No bootstrap required, opened a new log
I20260620 22:52:25.359845 10408 consensus_queue.cc:260] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.359866 10410 ts_tablet_manager.cc:1397] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.359939 10408 ts_tablet_manager.cc:1428] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:25.360170 10410 raft_consensus.cc:348] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.360249 10410 raft_consensus.cc:374] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.360270 10410 raft_consensus.cc:729] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e612d19296534eb18d1ac3e715d3deb3, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.360316 10410 consensus_queue.cc:260] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.360401 10410 ts_tablet_manager.cc:1428] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.360952 10409 tablet_bootstrap.cc:492] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: No bootstrap required, opened a new log
I20260620 22:52:25.361001 10409 ts_tablet_manager.cc:1397] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:52:25.361135 10409 raft_consensus.cc:348] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.361202 10409 raft_consensus.cc:374] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.361227 10409 raft_consensus.cc:729] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.361274 10409 consensus_queue.cc:260] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.361361 10409 ts_tablet_manager.cc:1428] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260620 22:52:25.365150 10384 tablet.cc:2369] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:25.425961 10414 raft_consensus.cc:482] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:25.426054 10414 raft_consensus.cc:504] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.426334 10414 leader_election.cc:288] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075), 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:25.429664 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.426761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:42563 (local address 127.2.90.130:42993)
0620 22:52:25.426798 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.426800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.426887 (+ 87us) server_negotiation.cc:407] Connection header received
0620 22:52:25.426951 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.426952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.426961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.427006 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.427204 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.427834 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.428916 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.429320 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.429390 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.429399 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.429565 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.429597 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:25.429694 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.426652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42993 (local address 127.2.90.131:42563)
0620 22:52:25.426810 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.426820 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.426885 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.427038 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.427039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.427041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.427130 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.427132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.427906 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.427907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.428835 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.428837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.429420 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.429421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.429491 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.429496 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.429556 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.429595 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260620 22:52:25.430037 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.426541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36075 (local address 127.2.90.131:48097)
0620 22:52:25.426596 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.426608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.426669 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.426842 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.426843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.426844 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.426909 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.426911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.427790 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.427791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.429108 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.429111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.429797 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.429798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.429875 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.429878 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.429942 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.429979 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:52:25.430099 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.426615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:48097 (local address 127.2.90.129:36075)
0620 22:52:25.426651 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.426652 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.426662 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:25.426744 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.426746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.426755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.426798 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.426983 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.427690 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.429198 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.429680 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.429744 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.429766 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.429989 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.430033 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:52:25.430042 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e3cf443d874bb8ac53ae3da35a8d85" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61" is_pre_election: true
I20260620 22:52:25.430166 10197 raft_consensus.cc:2454] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 0.
I20260620 22:52:25.430289 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e3cf443d874bb8ac53ae3da35a8d85" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3" is_pre_election: true
I20260620 22:52:25.430366 10046 raft_consensus.cc:2454] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 0.
I20260620 22:52:25.430382 10268 leader_election.cc:302] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, 8864618b57b344cf882362bde1a7ab61; no voters:
I20260620 22:52:25.430509 10414 raft_consensus.cc:2781] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:25.430546 10414 raft_consensus.cc:482] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:25.430563 10414 raft_consensus.cc:3037] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.431293 10414 raft_consensus.cc:504] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.431398 10414 leader_election.cc:288] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 election: Requested vote from peers e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075), 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:25.431561 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e3cf443d874bb8ac53ae3da35a8d85" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3"
I20260620 22:52:25.431604 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73e3cf443d874bb8ac53ae3da35a8d85" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61"
I20260620 22:52:25.431632 10046 raft_consensus.cc:3037] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.431658 10197 raft_consensus.cc:3037] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.432353 10046 raft_consensus.cc:2454] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 1.
I20260620 22:52:25.432351 10197 raft_consensus.cc:2454] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 1.
I20260620 22:52:25.432536 10268 leader_election.cc:302] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, e612d19296534eb18d1ac3e715d3deb3; no voters:
I20260620 22:52:25.432615 10414 raft_consensus.cc:2781] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:25.432801 10414 raft_consensus.cc:686] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 1 LEADER]: Becoming Leader. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Running, Role: LEADER
I20260620 22:52:25.432873 10414 consensus_queue.cc:237] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.433511 9899 catalog_manager.cc:5495] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 59cc15518a7443e3838631a8eaa249ca (127.2.90.131). New cstate: current_term: 1 leader_uuid: "59cc15518a7443e3838631a8eaa249ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } health_report { overall_health: HEALTHY } } }
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995944, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.190@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:25.487164 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.481343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.2.90.190:39549)
0620 22:52:25.481420 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.481422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.481488 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:52:25.481552 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.481553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.481563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.481610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.482040 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.482519 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.483168 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.483342 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.483393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.485245 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:25.485246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:25.485246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:25.485252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:25.486262 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.486605 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.486605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.486606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.486656 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.486799 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.486799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.486800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.486885 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:25.486927 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.487035 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.487098 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 22:52:25.487258 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.481276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39549 (local address 127.0.0.1:33860)
0620 22:52:25.481416 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.481427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.481497 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.481622 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.481623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.481923 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:25.481966 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.481969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.482588 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.482589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.483100 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.483102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.483399 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.483400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.483455 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.483880 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:25.483884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:25.485174 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:25.486326 (+ 1152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:25.486331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:25.486333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:25.486544 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:25.486708 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:25.486708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:25.486709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:25.486763 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:25.486924 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:25.486930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:25.486973 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.487019 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.487155 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":55,"thread_start_us":74,"threads_started":1}
I20260620 22:52:25.488934 9899 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33860:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 22:52:25.489106 9899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:25.490273 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb55bbf9f1547c49c0f142bdd88d09e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ef1335c210444fbaeb3438cf8fea896]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:52:25.490293 10317 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb55bbf9f1547c49c0f142bdd88d09e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ef1335c210444fbaeb3438cf8fea896]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:52:25.490288 10037 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb55bbf9f1547c49c0f142bdd88d09e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ef1335c210444fbaeb3438cf8fea896]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 22:52:25.490407 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb55bbf9f1547c49c0f142bdd88d09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.490424 10037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb55bbf9f1547c49c0f142bdd88d09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.490429 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb55bbf9f1547c49c0f142bdd88d09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.491801 10410 tablet_bootstrap.cc:492] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: Bootstrap starting.
I20260620 22:52:25.491819 10408 tablet_bootstrap.cc:492] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: Bootstrap starting.
I20260620 22:52:25.492120 10408 tablet_bootstrap.cc:654] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.492168 10410 tablet_bootstrap.cc:654] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.492444 10409 tablet_bootstrap.cc:492] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: Bootstrap starting.
I20260620 22:52:25.492609 10408 tablet_bootstrap.cc:492] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: No bootstrap required, opened a new log
I20260620 22:52:25.492640 10408 ts_tablet_manager.cc:1397] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.492781 10409 tablet_bootstrap.cc:654] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.492775 10408 raft_consensus.cc:348] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.492826 10408 raft_consensus.cc:374] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.492838 10408 raft_consensus.cc:729] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8864618b57b344cf882362bde1a7ab61, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.492871 10408 consensus_queue.cc:260] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.492951 10408 ts_tablet_manager.cc:1428] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.493005 10418 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.493098 10418 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.493271 10410 tablet_bootstrap.cc:492] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: No bootstrap required, opened a new log
I20260620 22:52:25.493323 10410 ts_tablet_manager.cc:1397] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:25.493467 10410 raft_consensus.cc:348] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.493539 10410 raft_consensus.cc:374] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.493558 10410 raft_consensus.cc:729] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e612d19296534eb18d1ac3e715d3deb3, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.493602 10410 consensus_queue.cc:260] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.493636 10409 tablet_bootstrap.cc:492] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: No bootstrap required, opened a new log
I20260620 22:52:25.493672 10409 ts_tablet_manager.cc:1397] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.493690 10410 ts_tablet_manager.cc:1428] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.493772 10415 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.493805 10409 raft_consensus.cc:348] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.493860 10409 raft_consensus.cc:374] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.493872 10409 raft_consensus.cc:729] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.493866 10415 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.493919 10409 consensus_queue.cc:260] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.494006 10409 ts_tablet_manager.cc:1428] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.494135 10414 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.494200 10414 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.660326 10415 raft_consensus.cc:482] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:25.660421 10415 raft_consensus.cc:504] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.660714 10415 leader_election.cc:288] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993), 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825)
I20260620 22:52:25.664527 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.660942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:40075 (local address 127.2.90.130:42993)
0620 22:52:25.660995 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.660997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.661128 (+ 131us) server_negotiation.cc:407] Connection header received
0620 22:52:25.661212 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.661213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.661229 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.661294 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.661445 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.662183 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.663669 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.664187 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.664261 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.664269 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.664412 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.664459 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:52:25.664522 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.660867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42993 (local address 127.2.90.129:40075)
0620 22:52:25.661065 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.661075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.661138 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.661302 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.661303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.661305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.661383 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.661384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.662255 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.662255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.663591 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.663593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.664264 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.664265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.664337 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.664342 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.664393 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.664434 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":77,"thread_start_us":116,"threads_started":1}
I20260620 22:52:25.664579 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.660963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:53125 (local address 127.2.90.131:37825)
0620 22:52:25.661010 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.661012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.661218 (+ 206us) server_negotiation.cc:407] Connection header received
0620 22:52:25.661248 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.661249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.661261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.661311 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:25.661449 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.662179 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.663786 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.664289 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.664342 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.664347 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.664490 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.664525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:52:25.664733 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.660884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:37825 (local address 127.2.90.129:53125)
0620 22:52:25.661157 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.661166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.661213 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.661311 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.661311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.661312 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:25.661382 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.661384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.662253 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.662254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.663590 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.663593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.664351 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.664352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.664421 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.664423 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.664468 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.664497 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":50,"thread_start_us":103,"threads_started":1}
I20260620 22:52:25.664829 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb55bbf9f1547c49c0f142bdd88d09e" candidate_uuid: "e612d19296534eb18d1ac3e715d3deb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61" is_pre_election: true
I20260620 22:52:25.664916 10197 raft_consensus.cc:2454] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e612d19296534eb18d1ac3e715d3deb3 in term 0.
I20260620 22:52:25.665009 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb55bbf9f1547c49c0f142bdd88d09e" candidate_uuid: "e612d19296534eb18d1ac3e715d3deb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cc15518a7443e3838631a8eaa249ca" is_pre_election: true
I20260620 22:52:25.665092 10337 raft_consensus.cc:2454] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e612d19296534eb18d1ac3e715d3deb3 in term 0.
I20260620 22:52:25.665076 9988 leader_election.cc:302] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8864618b57b344cf882362bde1a7ab61, e612d19296534eb18d1ac3e715d3deb3; no voters:
I20260620 22:52:25.665201 10415 raft_consensus.cc:2781] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:25.665246 10415 raft_consensus.cc:482] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:25.665271 10415 raft_consensus.cc:3037] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.665897 10415 raft_consensus.cc:504] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.666131 10415 leader_election.cc:288] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [CANDIDATE]: Term 1 election: Requested vote from peers 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993), 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825)
I20260620 22:52:25.666263 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb55bbf9f1547c49c0f142bdd88d09e" candidate_uuid: "e612d19296534eb18d1ac3e715d3deb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61"
I20260620 22:52:25.666343 10197 raft_consensus.cc:3037] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.666326 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb55bbf9f1547c49c0f142bdd88d09e" candidate_uuid: "e612d19296534eb18d1ac3e715d3deb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cc15518a7443e3838631a8eaa249ca"
I20260620 22:52:25.666390 10337 raft_consensus.cc:3037] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.667053 10197 raft_consensus.cc:2454] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e612d19296534eb18d1ac3e715d3deb3 in term 1.
I20260620 22:52:25.667053 10337 raft_consensus.cc:2454] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e612d19296534eb18d1ac3e715d3deb3 in term 1.
I20260620 22:52:25.667268 9988 leader_election.cc:302] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8864618b57b344cf882362bde1a7ab61, e612d19296534eb18d1ac3e715d3deb3; no voters:
I20260620 22:52:25.667357 10415 raft_consensus.cc:2781] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:25.667526 10415 raft_consensus.cc:686] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [term 1 LEADER]: Becoming Leader. State: Replica: e612d19296534eb18d1ac3e715d3deb3, State: Running, Role: LEADER
I20260620 22:52:25.667598 10415 consensus_queue.cc:237] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.667913 10450 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: TxnStatusTablet state changed. Reason: New leader e612d19296534eb18d1ac3e715d3deb3. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.667973 10450 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:52:25.668159 10452 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 22:52:25.668263 10452 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 22:52:25.668274 9898 catalog_manager.cc:5495] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129). New cstate: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } health_report { overall_health: UNKNOWN } } }
I20260620 22:52:25.675269 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.672581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36075 (local address 127.0.0.1:38724)
0620 22:52:25.672650 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.672662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.672739 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.672885 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.672886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.672887 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:25.672935 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.672937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.673533 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.673534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.674431 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.674435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.674772 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.674773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.674839 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.674855 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.675066 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.675072 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.675144 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.675193 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 22:52:25.675448 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.672730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.2.90.129:36075)
0620 22:52:25.672769 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.672771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.672782 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:25.672812 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.672813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.672828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.672879 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:25.672995 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.673465 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.674521 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.674698 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.674750 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.674932 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.675015 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.675060 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.675370 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.675389 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:25.676357 10337 raft_consensus.cc:1261] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Refusing update from remote peer e612d19296534eb18d1ac3e715d3deb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.676614 10197 raft_consensus.cc:1261] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Refusing update from remote peer e612d19296534eb18d1ac3e715d3deb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.676625 10450 consensus_queue.cc:1035] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.676841 10450 consensus_queue.cc:1035] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.677160 10414 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: TxnStatusTablet state changed. Reason: New leader e612d19296534eb18d1ac3e715d3deb3. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.677246 10414 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.677420 10418 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: TxnStatusTablet state changed. Reason: New leader e612d19296534eb18d1ac3e715d3deb3. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.677498 10418 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.678076 10422 mvcc.cc:205] Tried to move back new op lower bound from 7299055393488633856 to 7299055393454809088. Current Snapshot: MvccSnapshot[applied={T|T < 7299055393488633856}]
I20260620 22:52:25.678231 10415 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.678236 10450 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.678318 10415 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:52:25.678330 10450 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P e612d19296534eb18d1ac3e715d3deb3: This TxnStatusTablet replica's current role is: LEADER
I20260620 22:52:25.678756 10421 mvcc.cc:205] Tried to move back new op lower bound from 7299055393488633856 to 7299055393454809088. Current Snapshot: MvccSnapshot[applied={T|T < 7299055393488633856}]
I20260620 22:52:25.678752 10423 mvcc.cc:205] Tried to move back new op lower bound from 7299055393488633856 to 7299055393454809088. Current Snapshot: MvccSnapshot[applied={T|T < 7299055393488633856}]
I20260620 22:52:25.678820 10414 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.678834 10418 tablet_replica.cc:439] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e612d19296534eb18d1ac3e715d3deb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } } }
I20260620 22:52:25.678893 10414 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P 59cc15518a7443e3838631a8eaa249ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.678903 10418 tablet_replica.cc:442] T 3eb55bbf9f1547c49c0f142bdd88d09e P 8864618b57b344cf882362bde1a7ab61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 22:52:25.682000 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.679936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.2.90.131:37825)
0620 22:52:25.679974 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.679976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.679983 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:25.680030 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.680030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.680046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.680097 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:25.680192 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.680638 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.681327 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.681527 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.681576 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.681710 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.681809 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.681866 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.681909 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.681925 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:25.687278 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.684044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36075 (local address 127.2.90.131:37509)
0620 22:52:25.684770 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.684779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.684830 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.684928 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.684929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.684930 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:25.684976 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.684977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.685824 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.685825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.686369 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.686370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.686925 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.686926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.686994 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.687006 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.687163 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.687164 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.687192 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.687223 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":100,"threads_started":1}
I20260620 22:52:25.687283 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.684098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37509 (local address 127.2.90.129:36075)
0620 22:52:25.684129 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.684131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.684835 (+ 704us) server_negotiation.cc:407] Connection header received
0620 22:52:25.684861 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.684862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.684871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.684929 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:25.685011 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.685759 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.686450 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.686759 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.686829 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.687077 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.687111 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.687161 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.687196 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.687216 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:52:25.689388 10197 raft_consensus.cc:1261] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Refusing update from remote peer 59cc15518a7443e3838631a8eaa249ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.689471 10046 raft_consensus.cc:1261] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Refusing update from remote peer 59cc15518a7443e3838631a8eaa249ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.689800 10437 consensus_queue.cc:1035] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.689899 10414 consensus_queue.cc:1035] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.697916 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.695276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:58839 (local address 127.2.90.131:37825)
0620 22:52:25.695313 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.695314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.695336 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:52:25.695355 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.695356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.695364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.695399 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:25.695543 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.696099 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.697217 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.697478 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.697531 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.697681 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.697713 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.697755 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.697825 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.697843 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:52:25.697950 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.695109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:37825 (local address 127.2.90.129:58839)
0620 22:52:25.695232 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.695243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:25.695297 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:25.695410 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:25.695411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:25.695412 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 22:52:25.695492 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.695494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.696247 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.696249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.697154 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:25.697156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.697537 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.697538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:25.697602 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.697617 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.697766 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.697768 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:25.697819 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:52:25.697859 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260620 22:52:25.706363 9899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.706470 9899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.707089 9899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Sending DeleteTablet for 3 replicas of tablet 73e3cf443d874bb8ac53ae3da35a8d85
I20260620 22:52:25.707397 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e3cf443d874bb8ac53ae3da35a8d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:60824
I20260620 22:52:25.707494 10432 tablet_replica.cc:330] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: stopping tablet replica
I20260620 22:52:25.707545 10432 raft_consensus.cc:2229] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.707576 10432 raft_consensus.cc:2258] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.707698 10037 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e3cf443d874bb8ac53ae3da35a8d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:38722
I20260620 22:52:25.707772 10431 tablet_replica.cc:330] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: stopping tablet replica
I20260620 22:52:25.707814 10431 raft_consensus.cc:2229] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.707844 10431 raft_consensus.cc:2258] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.708189 10432 ts_tablet_manager.cc:1905] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.709867 10432 ts_tablet_manager.cc:1918] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:25.709920 10432 log.cc:1196] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/73e3cf443d874bb8ac53ae3da35a8d85
I20260620 22:52:25.710112 10432 ts_tablet_manager.cc:1939] T 73e3cf443d874bb8ac53ae3da35a8d85 P 8864618b57b344cf882362bde1a7ab61: Deleting consensus metadata
I20260620 22:52:25.710548 9884 catalog_manager.cc:4841] TS 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993): tablet 73e3cf443d874bb8ac53ae3da35a8d85 (table test-table [id=9c3abcfa476f475fb0cd3d9f971b3fe9]) successfully deleted
I20260620 22:52:25.711351 10317 tablet_service.cc:1514] Processing DeleteTablet for tablet 73e3cf443d874bb8ac53ae3da35a8d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:35562
I20260620 22:52:25.711448 10430 tablet_replica.cc:330] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: stopping tablet replica
I20260620 22:52:25.711499 10430 raft_consensus.cc:2229] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:25.711588 10430 raft_consensus.cc:2258] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.711645 10431 ts_tablet_manager.cc:1905] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.711862 10430 ts_tablet_manager.cc:1905] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.713215 10431 ts_tablet_manager.cc:1918] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:25.713271 10431 log.cc:1196] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/73e3cf443d874bb8ac53ae3da35a8d85
I20260620 22:52:25.713512 10431 ts_tablet_manager.cc:1939] T 73e3cf443d874bb8ac53ae3da35a8d85 P e612d19296534eb18d1ac3e715d3deb3: Deleting consensus metadata
I20260620 22:52:25.713698 10430 ts_tablet_manager.cc:1918] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:25.713747 10430 log.cc:1196] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/73e3cf443d874bb8ac53ae3da35a8d85
I20260620 22:52:25.713976 10430 ts_tablet_manager.cc:1939] T 73e3cf443d874bb8ac53ae3da35a8d85 P 59cc15518a7443e3838631a8eaa249ca: Deleting consensus metadata
I20260620 22:52:25.714291 9884 catalog_manager.cc:4841] TS e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075): tablet 73e3cf443d874bb8ac53ae3da35a8d85 (table test-table [id=9c3abcfa476f475fb0cd3d9f971b3fe9]) successfully deleted
I20260620 22:52:25.714632 9887 catalog_manager.cc:4841] TS 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825): tablet 73e3cf443d874bb8ac53ae3da35a8d85 (table test-table [id=9c3abcfa476f475fb0cd3d9f971b3fe9]) successfully deleted
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:52:25.745283 9949 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:25.738575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.2.90.190:39549)
0620 22:52:25.738619 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.738621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.738779 (+ 158us) server_negotiation.cc:407] Connection header received
0620 22:52:25.738838 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.738839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.738851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.738913 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.739592 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.740348 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.741484 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.741809 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.741880 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.745212 (+ 3332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33874: BlockingRecv error: recv got EOF from 127.0.0.1:33874 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:52:25 dist-test-slave-4mtw krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM
I20260620 22:52:25.754902 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.746294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.2.90.190:39549)
0620 22:52:25.746332 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.746334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.746555 (+ 221us) server_negotiation.cc:407] Connection header received
0620 22:52:25.746604 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.746605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.746615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.746674 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:25.747222 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.747931 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.748990 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.749263 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.749337 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.751882 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:25.751883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:25.751884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:25.751892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:25.753666 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.754047 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.754048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.754049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.754108 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:25.754352 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:25.754353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:25.754353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:25.754484 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:25.754580 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.754723 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.754804 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:25.756668 9899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:25.756886 9899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:25.758212 10037 tablet_service.cc:1467] Processing CreateTablet for tablet fd25046c027a45cc84979defb53bdddf (DEFAULT_TABLE table=test-table [id=fde81b4967da40ab9579f729229ebd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.758234 10317 tablet_service.cc:1467] Processing CreateTablet for tablet fd25046c027a45cc84979defb53bdddf (DEFAULT_TABLE table=test-table [id=fde81b4967da40ab9579f729229ebd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.758338 10037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd25046c027a45cc84979defb53bdddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.758359 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd25046c027a45cc84979defb53bdddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.759640 10410 tablet_bootstrap.cc:492] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Bootstrap starting.
I20260620 22:52:25.759709 10409 tablet_bootstrap.cc:492] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Bootstrap starting.
I20260620 22:52:25.760022 10410 tablet_bootstrap.cc:654] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.760098 10409 tablet_bootstrap.cc:654] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.760397 10177 tablet_service.cc:1467] Processing CreateTablet for tablet fd25046c027a45cc84979defb53bdddf (DEFAULT_TABLE table=test-table [id=fde81b4967da40ab9579f729229ebd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.760499 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd25046c027a45cc84979defb53bdddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.761898 10408 tablet_bootstrap.cc:492] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Bootstrap starting.
I20260620 22:52:25.762169 10408 tablet_bootstrap.cc:654] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.762693 10408 tablet_bootstrap.cc:492] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: No bootstrap required, opened a new log
I20260620 22:52:25.762737 10408 ts_tablet_manager.cc:1397] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.762867 10408 raft_consensus.cc:348] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.762928 10408 raft_consensus.cc:374] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.762946 10408 raft_consensus.cc:729] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8864618b57b344cf882362bde1a7ab61, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.762988 10408 consensus_queue.cc:260] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.763068 10408 ts_tablet_manager.cc:1428] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.763880 10410 tablet_bootstrap.cc:492] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: No bootstrap required, opened a new log
I20260620 22:52:25.763927 10410 ts_tablet_manager.cc:1397] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 22:52:25.764057 10410 raft_consensus.cc:348] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.764117 10410 raft_consensus.cc:374] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.764134 10410 raft_consensus.cc:729] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e612d19296534eb18d1ac3e715d3deb3, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.764175 10410 consensus_queue.cc:260] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.764253 10410 ts_tablet_manager.cc:1428] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.764729 10409 tablet_bootstrap.cc:492] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: No bootstrap required, opened a new log
I20260620 22:52:25.764786 10409 ts_tablet_manager.cc:1397] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 22:52:25.764892 10409 raft_consensus.cc:348] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.764947 10409 raft_consensus.cc:374] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.764966 10409 raft_consensus.cc:729] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.765002 10409 consensus_queue.cc:260] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.765079 10409 ts_tablet_manager.cc:1428] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.869096 10437 raft_consensus.cc:482] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:25.869180 10437 raft_consensus.cc:504] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.869324 10437 leader_election.cc:288] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075), 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:25.869515 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25046c027a45cc84979defb53bdddf" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3" is_pre_election: true
I20260620 22:52:25.869598 10046 raft_consensus.cc:2454] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 0.
I20260620 22:52:25.869536 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25046c027a45cc84979defb53bdddf" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61" is_pre_election: true
I20260620 22:52:25.869638 10197 raft_consensus.cc:2454] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 0.
I20260620 22:52:25.869796 10268 leader_election.cc:302] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, 8864618b57b344cf882362bde1a7ab61; no voters:
I20260620 22:52:25.869876 10437 raft_consensus.cc:2781] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:25.869925 10437 raft_consensus.cc:482] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:25.869952 10437 raft_consensus.cc:3037] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.870680 10437 raft_consensus.cc:504] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.870784 10437 leader_election.cc:288] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 election: Requested vote from peers e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075), 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:25.870950 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25046c027a45cc84979defb53bdddf" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61"
I20260620 22:52:25.870944 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25046c027a45cc84979defb53bdddf" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3"
I20260620 22:52:25.871025 10197 raft_consensus.cc:3037] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.871025 10046 raft_consensus.cc:3037] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.871794 10197 raft_consensus.cc:2454] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 1.
I20260620 22:52:25.871791 10046 raft_consensus.cc:2454] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 1.
I20260620 22:52:25.871942 10268 leader_election.cc:302] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, 8864618b57b344cf882362bde1a7ab61; no voters:
I20260620 22:52:25.872030 10437 raft_consensus.cc:2781] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:25.872088 10437 raft_consensus.cc:686] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 1 LEADER]: Becoming Leader. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Running, Role: LEADER
I20260620 22:52:25.872135 10437 consensus_queue.cc:237] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.872720 9898 catalog_manager.cc:5495] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 59cc15518a7443e3838631a8eaa249ca (127.2.90.131). New cstate: current_term: 1 leader_uuid: "59cc15518a7443e3838631a8eaa249ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } health_report { overall_health: HEALTHY } } }
I20260620 22:52:25.880064 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.878283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.2.90.131:37825)
0620 22:52:25.878320 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:25.878322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:25.878330 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:25.878371 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:25.878371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:25.878379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:25.878436 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:25.878519 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.878954 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.879628 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:25.879793 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:25.879833 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:25.879916 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:25.879938 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:25.879963 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:25.879999 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 22:52:25.880014 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:52:25.880651 10197 raft_consensus.cc:1261] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Refusing update from remote peer 59cc15518a7443e3838631a8eaa249ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.880669 10046 raft_consensus.cc:1261] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Refusing update from remote peer 59cc15518a7443e3838631a8eaa249ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.880872 10465 consensus_queue.cc:1035] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.880985 10437 consensus_queue.cc:1035] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.883211 9898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.883286 9898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.883705 9898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Sending DeleteTablet for 3 replicas of tablet fd25046c027a45cc84979defb53bdddf
I20260620 22:52:25.883908 10037 tablet_service.cc:1514] Processing DeleteTablet for tablet fd25046c027a45cc84979defb53bdddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:38722
I20260620 22:52:25.883917 10317 tablet_service.cc:1514] Processing DeleteTablet for tablet fd25046c027a45cc84979defb53bdddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:35562
I20260620 22:52:25.883960 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet fd25046c027a45cc84979defb53bdddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:60824
I20260620 22:52:25.884006 10431 tablet_replica.cc:330] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: stopping tablet replica
I20260620 22:52:25.884006 10430 tablet_replica.cc:330] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: stopping tablet replica
I20260620 22:52:25.884037 10432 tablet_replica.cc:330] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: stopping tablet replica
I20260620 22:52:25.884049 10431 raft_consensus.cc:2229] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.884049 10430 raft_consensus.cc:2229] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:25.884087 10431 raft_consensus.cc:2258] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.884083 10432 raft_consensus.cc:2229] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.884097 10430 raft_consensus.cc:2258] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.884122 10432 raft_consensus.cc:2258] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.884393 10432 ts_tablet_manager.cc:1905] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.884447 10430 ts_tablet_manager.cc:1905] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.884534 10431 ts_tablet_manager.cc:1905] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.885896 9898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:25.886039 9898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:25.886080 10430 ts_tablet_manager.cc:1918] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:25.886080 10431 ts_tablet_manager.cc:1918] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:25.886129 10430 log.cc:1196] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/fd25046c027a45cc84979defb53bdddf
I20260620 22:52:25.886132 10431 log.cc:1196] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/fd25046c027a45cc84979defb53bdddf
I20260620 22:52:25.886330 10431 ts_tablet_manager.cc:1939] T fd25046c027a45cc84979defb53bdddf P e612d19296534eb18d1ac3e715d3deb3: Deleting consensus metadata
I20260620 22:52:25.886343 10430 ts_tablet_manager.cc:1939] T fd25046c027a45cc84979defb53bdddf P 59cc15518a7443e3838631a8eaa249ca: Deleting consensus metadata
I20260620 22:52:25.886626 9887 catalog_manager.cc:4841] TS 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825): tablet fd25046c027a45cc84979defb53bdddf (table test-table [id=fde81b4967da40ab9579f729229ebd9a]) successfully deleted
I20260620 22:52:25.886673 9884 catalog_manager.cc:4841] TS e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075): tablet fd25046c027a45cc84979defb53bdddf (table test-table [id=fde81b4967da40ab9579f729229ebd9a]) successfully deleted
I20260620 22:52:25.886868 10432 ts_tablet_manager.cc:1918] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:25.886929 10432 log.cc:1196] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/fd25046c027a45cc84979defb53bdddf
I20260620 22:52:25.887108 10432 ts_tablet_manager.cc:1939] T fd25046c027a45cc84979defb53bdddf P 8864618b57b344cf882362bde1a7ab61: Deleting consensus metadata
I20260620 22:52:25.887527 10317 tablet_service.cc:1467] Processing CreateTablet for tablet 7e21cfdcaece493b8138b93e662e539b (DEFAULT_TABLE table=test-table [id=6e5fc608bea1490fae3985b30865a898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.887553 9884 catalog_manager.cc:4841] TS 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993): tablet fd25046c027a45cc84979defb53bdddf (table test-table [id=fde81b4967da40ab9579f729229ebd9a]) successfully deleted
I20260620 22:52:25.887537 10037 tablet_service.cc:1467] Processing CreateTablet for tablet 7e21cfdcaece493b8138b93e662e539b (DEFAULT_TABLE table=test-table [id=6e5fc608bea1490fae3985b30865a898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.887624 10317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e21cfdcaece493b8138b93e662e539b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.887612 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 7e21cfdcaece493b8138b93e662e539b (DEFAULT_TABLE table=test-table [id=6e5fc608bea1490fae3985b30865a898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:25.887633 10037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e21cfdcaece493b8138b93e662e539b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.887683 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e21cfdcaece493b8138b93e662e539b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:25.888401 10410 tablet_bootstrap.cc:492] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: Bootstrap starting.
I20260620 22:52:25.888716 10409 tablet_bootstrap.cc:492] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Bootstrap starting.
I20260620 22:52:25.888718 10410 tablet_bootstrap.cc:654] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.888720 10408 tablet_bootstrap.cc:492] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Bootstrap starting.
I20260620 22:52:25.888994 10409 tablet_bootstrap.cc:654] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.889012 10408 tablet_bootstrap.cc:654] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:25.889447 10409 tablet_bootstrap.cc:492] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: No bootstrap required, opened a new log
I20260620 22:52:25.889449 10408 tablet_bootstrap.cc:492] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: No bootstrap required, opened a new log
I20260620 22:52:25.889492 10409 ts_tablet_manager.cc:1397] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:25.889492 10408 ts_tablet_manager.cc:1397] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:52:25.889585 10409 raft_consensus.cc:348] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.889586 10408 raft_consensus.cc:348] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.889631 10410 tablet_bootstrap.cc:492] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: No bootstrap required, opened a new log
I20260620 22:52:25.889640 10408 raft_consensus.cc:374] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.889640 10409 raft_consensus.cc:374] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.889662 10410 ts_tablet_manager.cc:1397] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:25.889662 10408 raft_consensus.cc:729] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8864618b57b344cf882362bde1a7ab61, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.889662 10409 raft_consensus.cc:729] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.889704 10408 consensus_queue.cc:260] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.889709 10409 consensus_queue.cc:260] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.889752 10410 raft_consensus.cc:348] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.889787 10408 ts_tablet_manager.cc:1428] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.889786 10409 ts_tablet_manager.cc:1428] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.889810 10410 raft_consensus.cc:374] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:25.889828 10410 raft_consensus.cc:729] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e612d19296534eb18d1ac3e715d3deb3, State: Initialized, Role: FOLLOWER
I20260620 22:52:25.889866 10410 consensus_queue.cc:260] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.889959 10410 ts_tablet_manager.cc:1428] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:25.934069 10414 raft_consensus.cc:482] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:25.934151 10414 raft_consensus.cc:504] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.934283 10414 leader_election.cc:288] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075), 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:25.934470 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e21cfdcaece493b8138b93e662e539b" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61" is_pre_election: true
I20260620 22:52:25.934470 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e21cfdcaece493b8138b93e662e539b" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3" is_pre_election: true
I20260620 22:52:25.934576 10046 raft_consensus.cc:2454] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 0.
I20260620 22:52:25.934576 10197 raft_consensus.cc:2454] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 0.
I20260620 22:52:25.934710 10268 leader_election.cc:302] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, 8864618b57b344cf882362bde1a7ab61; no voters:
I20260620 22:52:25.934801 10414 raft_consensus.cc:2781] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:25.934839 10414 raft_consensus.cc:482] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:25.934863 10414 raft_consensus.cc:3037] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.935436 10414 raft_consensus.cc:504] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.935534 10414 leader_election.cc:288] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 election: Requested vote from peers e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075), 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993)
I20260620 22:52:25.935672 10046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e21cfdcaece493b8138b93e662e539b" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e612d19296534eb18d1ac3e715d3deb3"
I20260620 22:52:25.935684 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e21cfdcaece493b8138b93e662e539b" candidate_uuid: "59cc15518a7443e3838631a8eaa249ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8864618b57b344cf882362bde1a7ab61"
I20260620 22:52:25.935747 10046 raft_consensus.cc:3037] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.935747 10197 raft_consensus.cc:3037] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:25.936429 10197 raft_consensus.cc:2454] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 1.
I20260620 22:52:25.936439 10046 raft_consensus.cc:2454] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59cc15518a7443e3838631a8eaa249ca in term 1.
I20260620 22:52:25.936568 10268 leader_election.cc:302] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cc15518a7443e3838631a8eaa249ca, e612d19296534eb18d1ac3e715d3deb3; no voters:
I20260620 22:52:25.936651 10414 raft_consensus.cc:2781] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:25.936720 10414 raft_consensus.cc:686] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 1 LEADER]: Becoming Leader. State: Replica: 59cc15518a7443e3838631a8eaa249ca, State: Running, Role: LEADER
I20260620 22:52:25.936767 10414 consensus_queue.cc:237] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } }
I20260620 22:52:25.937258 9899 catalog_manager.cc:5495] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 59cc15518a7443e3838631a8eaa249ca (127.2.90.131). New cstate: current_term: 1 leader_uuid: "59cc15518a7443e3838631a8eaa249ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59cc15518a7443e3838631a8eaa249ca" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 37825 } health_report { overall_health: HEALTHY } } }
I20260620 22:52:25.948905 10046 raft_consensus.cc:1261] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Refusing update from remote peer 59cc15518a7443e3838631a8eaa249ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.949041 10197 raft_consensus.cc:1261] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Refusing update from remote peer 59cc15518a7443e3838631a8eaa249ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:25.949121 10414 consensus_queue.cc:1035] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e612d19296534eb18d1ac3e715d3deb3" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.949211 10465 consensus_queue.cc:1035] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "8864618b57b344cf882362bde1a7ab61" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:25.957760 9899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.957840 9899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:25.958348 9899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f26ade53224579912758e8c5773c0d: Sending DeleteTablet for 3 replicas of tablet 7e21cfdcaece493b8138b93e662e539b
I20260620 22:52:25.958535 10037 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e21cfdcaece493b8138b93e662e539b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:38722
I20260620 22:52:25.958595 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e21cfdcaece493b8138b93e662e539b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:60824
I20260620 22:52:25.958638 10431 tablet_replica.cc:330] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: stopping tablet replica
I20260620 22:52:25.958638 10317 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e21cfdcaece493b8138b93e662e539b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:25 UTC) from {username='oryx', principal='oryx/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:35562
I20260620 22:52:25.958673 10432 tablet_replica.cc:330] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: stopping tablet replica
I20260620 22:52:25.958691 10431 raft_consensus.cc:2229] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.958709 10432 raft_consensus.cc:2229] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:25.958714 10430 tablet_replica.cc:330] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: stopping tablet replica
I20260620 22:52:25.958722 10431 raft_consensus.cc:2258] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.958743 10432 raft_consensus.cc:2258] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.958751 10430 raft_consensus.cc:2229] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:52:25.958791 10430 raft_consensus.cc:2258] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:25.959043 10431 ts_tablet_manager.cc:1905] T 7e21cfdcaece493b8138b93e662e539b P e612d19296534eb18d1ac3e715d3deb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.959034 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 9972
I20260620 22:52:25.959137 10430 ts_tablet_manager.cc:1905] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.960481 10430 ts_tablet_manager.cc:1918] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:25.960532 10430 log.cc:1196] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/7e21cfdcaece493b8138b93e662e539b
I20260620 22:52:25.960749 10430 ts_tablet_manager.cc:1939] T 7e21cfdcaece493b8138b93e662e539b P 59cc15518a7443e3838631a8eaa249ca: Deleting consensus metadata
I20260620 22:52:25.961026 10432 ts_tablet_manager.cc:1905] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:25.961071 9887 catalog_manager.cc:4841] TS 59cc15518a7443e3838631a8eaa249ca (127.2.90.131:37825): tablet 7e21cfdcaece493b8138b93e662e539b (table test-table [id=6e5fc608bea1490fae3985b30865a898]) successfully deleted
I20260620 22:52:25.961992 10432 ts_tablet_manager.cc:1918] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 22:52:25.962036 10432 log.cc:1196] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/7e21cfdcaece493b8138b93e662e539b
I20260620 22:52:25.962190 10432 ts_tablet_manager.cc:1939] T 7e21cfdcaece493b8138b93e662e539b P 8864618b57b344cf882362bde1a7ab61: Deleting consensus metadata
I20260620 22:52:25.962448 9884 catalog_manager.cc:4841] TS 8864618b57b344cf882362bde1a7ab61 (127.2.90.130:42993): tablet 7e21cfdcaece493b8138b93e662e539b (table test-table [id=6e5fc608bea1490fae3985b30865a898]) successfully deleted
W20260620 22:52:25.964668 9884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:36075 (error 108)
I20260620 22:52:25.964861 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 10112
I20260620 22:52:25.965740 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:25.965554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36075 (local address 127.0.0.1:38730)
0620 22:52:25.965661 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:25.965704 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:36075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
W20260620 22:52:25.965857 9884 catalog_manager.cc:4584] TS e612d19296534eb18d1ac3e715d3deb3 (127.2.90.129:36075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e21cfdcaece493b8138b93e662e539b: Network error: Client connection negotiation failed: client connection to 127.2.90.129:36075: connect: Connection refused (error 111)
I20260620 22:52:25.970222 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 10252
I20260620 22:52:25.975278 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 9868
2026-06-20T22:52:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2503 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:26 dist-test-slave-4mtw krb5kdc[10497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:26 dist-test-slave-4mtw krb5kdc[10497](info): set up 2 sockets
Jun 20 22:52:26 dist-test-slave-4mtw krb5kdc[10497](info): commencing operation
krb5kdc: starting...
W20260620 22:52:26.921087 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:26 dist-test-slave-4mtw krb5kdc[10497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:27.027109 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:40007
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:27.104988 10510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:27.105191 10510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:27.105221 10510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:27.106388 10510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:27.106427 10510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:27.106446 10510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:27.106460 10510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:27.106477 10510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:27.108017 10510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:40007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:27.108258 10510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:27.108453 10510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:27.112879 10518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:27.112913 10510 server_base.cc:1034] running on GCE node
W20260620 22:52:27.112891 10516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:27.112891 10515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:27.113279 10510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:27.113546 10510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:27.114682 10510 hybrid_clock.cc:648] HybridClock initialized: now 1781995947114667 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T22:52:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1149 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:27 dist-test-slave-4mtw krb5kdc[10523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:27 dist-test-slave-4mtw krb5kdc[10523](info): set up 2 sockets
Jun 20 22:52:27 dist-test-slave-4mtw krb5kdc[10523](info): commencing operation
krb5kdc: starting...
W20260620 22:52:28.076889 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:28.180904 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35549
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:28.256959 10536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:28.257148 10536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:28.257169 10536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:28.258283 10536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:28.258325 10536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:28.258338 10536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:28.258350 10536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:28.258360 10536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:28.259788 10536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:28.260000 10536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:28.260201 10536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:28.264529 10542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.264621 10536 server_base.cc:1034] running on GCE node
W20260620 22:52:28.264544 10541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:28.264537 10544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.264953 10536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:28.265197 10536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:28.266327 10536 hybrid_clock.cc:648] HybridClock initialized: now 1781995948266317 us; error 28 us; skew 500 ppm
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:28.268479 10536 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:28.268903 10536 webserver.cc:460] Webserver started at http://127.2.90.190:33933/ using document root <none> and password file <none>
I20260620 22:52:28.269114 10536 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:28.269156 10536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:28.269275 10536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:28.270246 10536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "c2024c1f1b8b44318bb3e6dac1841eab"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "d06229c5bcde65b291bc3fad311e1f4c"
server_key_iv: "7deda463b1233fe65f81921c6be6ba02"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.270553 10536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "c2024c1f1b8b44318bb3e6dac1841eab"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "d06229c5bcde65b291bc3fad311e1f4c"
server_key_iv: "7deda463b1233fe65f81921c6be6ba02"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.271687 10536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:28.272270 10551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.272398 10536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:28.272477 10536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "c2024c1f1b8b44318bb3e6dac1841eab"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "d06229c5bcde65b291bc3fad311e1f4c"
server_key_iv: "7deda463b1233fe65f81921c6be6ba02"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.272535 10536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:28.284162 10536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:28.285784 10536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:28.285905 10536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:28.289752 10536 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35549
I20260620 22:52:28.289784 10603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35549 every 8 connection(s)
I20260620 22:52:28.290087 10536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:28.290812 10604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:28.293036 10604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Bootstrap starting.
I20260620 22:52:28.293524 10604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:28.293737 10604 log.cc:824] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:28.294327 10604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: No bootstrap required, opened a new log
I20260620 22:52:28.294435 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 10536
I20260620 22:52:28.294543 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:28.294725 2410 external_mini_cluster.cc:1428] Setting key fa4803ef96f44f98bb9615871b343566
I20260620 22:52:28.296162 10604 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.296301 10604 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:28.296334 10604 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2024c1f1b8b44318bb3e6dac1841eab, State: Initialized, Role: FOLLOWER
I20260620 22:52:28.296490 10604 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.296572 10604 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:28.296608 10604 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:28.296651 10604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:28.297557 10604 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.297641 10604 leader_election.cc:302] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2024c1f1b8b44318bb3e6dac1841eab; no voters:
I20260620 22:52:28.297775 10604 leader_election.cc:288] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:28.297817 10609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:28.297976 10609 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 LEADER]: Becoming Leader. State: Replica: c2024c1f1b8b44318bb3e6dac1841eab, State: Running, Role: LEADER
I20260620 22:52:28.298014 10604 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:28.298118 10609 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.298478 10610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } } }
I20260620 22:52:28.298552 10610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:28.298749 10611 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2024c1f1b8b44318bb3e6dac1841eab. Latest consensus state: current_term: 1 leader_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } } }
I20260620 22:52:28.299036 10611 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:28.299181 10617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:28.299612 10617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:28.301081 10617 catalog_manager.cc:1347] Generated new cluster ID: 5ab68ee4c7144eb3b1c6673a53a8a505
I20260620 22:52:28.301134 10617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:28.304950 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.295165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.2.90.190:35549)
0620 22:52:28.295319 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:28.295326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:28.295349 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:52:28.296136 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:28.296142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:28.296184 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:28.296287 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:28.297061 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.298183 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.299257 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.299835 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.299944 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.302166 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:28.302170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:28.302170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:28.302181 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:28.303590 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.303887 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.303890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.303891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.303994 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.304164 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.304165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.304165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.304385 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:28.304526 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:28.304622 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 22:52:28.304770 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260620 22:52:28.307545 10617 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:28.308029 10617 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:28.311859 10617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Generated new TSK 0
I20260620 22:52:28.312008 10617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:28.333899 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35549
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:28.410611 10630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:28.410794 10630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:28.410815 10630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:28.411917 10630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:28.411964 10630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:28.411989 10630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:28.413408 10630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:28.413614 10630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:28.413806 10630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:28.419057 10636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.419145 10630 server_base.cc:1034] running on GCE node
W20260620 22:52:28.419078 10638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:28.419209 10635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.419471 10630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:28.419699 10630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:28.420835 10630 hybrid_clock.cc:648] HybridClock initialized: now 1781995948420823 us; error 30 us; skew 500 ppm
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:28.423023 10630 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:28.423411 10630 webserver.cc:460] Webserver started at http://127.2.90.129:33757/ using document root <none> and password file <none>
I20260620 22:52:28.423604 10630 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:28.423645 10630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:28.423763 10630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:28.424793 10630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "cc486ed57d5d49679b7fc2afa24a69bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "226997c020a55a1c48b5f231a5780f31"
server_key_iv: "cb3ead3a8836dc13851375494f15c71d"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.425194 10630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "cc486ed57d5d49679b7fc2afa24a69bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "226997c020a55a1c48b5f231a5780f31"
server_key_iv: "cb3ead3a8836dc13851375494f15c71d"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.426501 10630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:28.427098 10645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.427223 10630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:28.427299 10630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "cc486ed57d5d49679b7fc2afa24a69bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "226997c020a55a1c48b5f231a5780f31"
server_key_iv: "cb3ead3a8836dc13851375494f15c71d"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.427358 10630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:28.449026 10630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:28.450865 10630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:28.451001 10630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:28.451205 10630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:28.451508 10630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:28.451546 10630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.451574 10630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:28.451599 10630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.456933 10630 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:45655
I20260620 22:52:28.456964 10758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:45655 every 8 connection(s)
I20260620 22:52:28.457260 10630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:28.457841 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 10630
I20260620 22:52:28.457952 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:28.458159 2410 external_mini_cluster.cc:1428] Setting key 0843bdea0a8f7036629fd81b8f52251b
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:52:28.466710 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.457883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:53225 (local address 127.2.90.190:35549)
0620 22:52:28.457932 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:28.457935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:28.458151 (+ 216us) server_negotiation.cc:407] Connection header received
0620 22:52:28.458735 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:28.458735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:28.458744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:28.458772 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:28.459892 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.460368 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.461735 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.461936 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.461988 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.464213 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:28.464214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:28.464214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:28.464221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:28.465464 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.465897 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.465898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.465898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.465943 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.466191 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.466192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.466192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.466288 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:28.466336 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:28.466585 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 22:52:28.466646 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:52:28.466885 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.457849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:53225)
0620 22:52:28.458009 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:28.458057 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:28.458650 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:28.458832 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:28.458835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:28.459544 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:28.459808 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:28.459814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.460446 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.460447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:28.461657 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:28.461660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.462007 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.462008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:28.462145 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.462596 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:28.462605 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:28.464128 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:28.465538 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:28.465544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:28.465546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:28.465832 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:28.465993 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:28.465994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:28.465994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:28.466126 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:28.466346 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:28.466352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:28.466481 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 22:52:28.466567 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:52:28.466684 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260620 22:52:28.467373 10759 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35549
I20260620 22:52:28.467453 10759 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:28.467597 10759 heartbeater.cc:511] Master 127.2.90.190:35549 requested a full tablet report, sending...
I20260620 22:52:28.468174 10568 ts_manager.cc:194] Registered new tserver with Master: cc486ed57d5d49679b7fc2afa24a69bb (127.2.90.129:45655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
I20260620 22:52:28.469004 10568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:53225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:28.482652 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35549
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:28.559451 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:28.559638 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:28.559657 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:28.560750 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:28.560781 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:28.560806 10764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:28.562119 10764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:28.562288 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:28.562454 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:28.566761 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:28.566807 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.566915 10764 server_base.cc:1034] running on GCE node
W20260620 22:52:28.566807 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.567127 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:28.567315 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:28.568454 10764 hybrid_clock.cc:648] HybridClock initialized: now 1781995948568443 us; error 27 us; skew 500 ppm
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:28.570677 10764 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:28.571039 10764 webserver.cc:460] Webserver started at http://127.2.90.130:36899/ using document root <none> and password file <none>
I20260620 22:52:28.571225 10764 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:28.571270 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:28.571388 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:28.572352 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "a6a51e31703b451e8a787ddfe47d81bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "ac8f4602837064ffc65e1182ecd4e8a0"
server_key_iv: "a6a05b1117ca57f951ea81b2b9cfa5ec"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.572669 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "a6a51e31703b451e8a787ddfe47d81bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "ac8f4602837064ffc65e1182ecd4e8a0"
server_key_iv: "a6a05b1117ca57f951ea81b2b9cfa5ec"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.573808 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:28.574391 10779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.574539 10764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:28.574622 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "a6a51e31703b451e8a787ddfe47d81bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "ac8f4602837064ffc65e1182ecd4e8a0"
server_key_iv: "a6a05b1117ca57f951ea81b2b9cfa5ec"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.574673 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:28.582124 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:28.583585 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:28.583683 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:28.583832 10764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:28.584057 10764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:28.584089 10764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.584107 10764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:28.584123 10764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.589255 10764 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45301
I20260620 22:52:28.589316 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45301 every 8 connection(s)
I20260620 22:52:28.589564 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:28.596825 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 10764
I20260620 22:52:28.596937 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:28.597131 2410 external_mini_cluster.cc:1428] Setting key 86a56c28a95a4ed5ec743ba8c6fec28a
I20260620 22:52:28.597563 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.590141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:36613 (local address 127.2.90.190:35549)
0620 22:52:28.590198 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:28.590200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:28.590363 (+ 163us) server_negotiation.cc:407] Connection header received
0620 22:52:28.590784 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:28.590785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:28.590794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:28.590842 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:28.591654 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.592196 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.593187 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.593367 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.593413 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.595164 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:28.595165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:28.595166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:28.595172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:28.596266 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.596618 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.596619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.596619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.596666 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.596939 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.596940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.596940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.597060 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:28.597110 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:28.597405 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 22:52:28.597485 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:52:28.597745 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.590117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:36613)
0620 22:52:28.590260 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:28.590292 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:28.590716 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:28.590893 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:28.590896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:28.591416 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:28.591590 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:28.591595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.592287 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.592287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:28.593128 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:28.593131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.593421 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.593422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:28.593511 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.593879 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:28.593886 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:28.595088 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:28.596329 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:28.596335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:28.596336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:28.596560 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:28.596738 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:28.596739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:28.596739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:28.596867 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:28.597118 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:28.597124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:28.597278 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:52:28.597371 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 22:52:28.597520 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 22:52:28.598101 10893 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35549
I20260620 22:52:28.598186 10893 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:28.598341 10893 heartbeater.cc:511] Master 127.2.90.190:35549 requested a full tablet report, sending...
I20260620 22:52:28.598681 10568 ts_manager.cc:194] Registered new tserver with Master: a6a51e31703b451e8a787ddfe47d81bb (127.2.90.130:45301)
I20260620 22:52:28.599114 10568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:36613
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:28.618698 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35549
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:28.696170 10898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:28.696363 10898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:28.696393 10898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:28.697593 10898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:28.697633 10898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:28.697669 10898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:28.699118 10898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:28.699344 10898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:28.699542 10898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:28.703969 10903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.704041 10898 server_base.cc:1034] running on GCE node
W20260620 22:52:28.703987 10904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:28.703976 10906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.704355 10898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:28.704583 10898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:28.705717 10898 hybrid_clock.cc:648] HybridClock initialized: now 1781995948705702 us; error 27 us; skew 500 ppm
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:28.707932 10898 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:28.708340 10898 webserver.cc:460] Webserver started at http://127.2.90.131:33671/ using document root <none> and password file <none>
I20260620 22:52:28.708547 10898 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:28.708590 10898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:28.708739 10898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:28.709712 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "0a42e2831ecf46089187d94c6f465b3c"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "bad941078b2407b72999f731c87a7563"
server_key_iv: "b982314c3a597a3f65bce661e7abf485"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.710024 10898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "0a42e2831ecf46089187d94c6f465b3c"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "bad941078b2407b72999f731c87a7563"
server_key_iv: "b982314c3a597a3f65bce661e7abf485"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.711143 10898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:52:28.711784 10913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.711933 10898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.712005 10898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "0a42e2831ecf46089187d94c6f465b3c"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "bad941078b2407b72999f731c87a7563"
server_key_iv: "b982314c3a597a3f65bce661e7abf485"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.712069 10898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:28.718086 10898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:28.719789 10898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:28.719897 10898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:28.720088 10898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:28.720352 10898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:28.720383 10898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.720409 10898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:28.720427 10898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.725760 10898 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:34201
I20260620 22:52:28.725796 11026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:34201 every 8 connection(s)
I20260620 22:52:28.726060 10898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:28.732590 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 10898
I20260620 22:52:28.732718 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:28.732937 2410 external_mini_cluster.cc:1428] Setting key 90f36b2da10e2d9d03b3dd1be2505f49
I20260620 22:52:28.734337 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.726661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:56093 (local address 127.2.90.190:35549)
0620 22:52:28.726708 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:28.726710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:28.726914 (+ 204us) server_negotiation.cc:407] Connection header received
0620 22:52:28.727385 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:28.727386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:28.727399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:28.727450 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:28.728324 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.728837 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.729768 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.729951 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.730024 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.731877 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:28.731879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:28.731879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:28.731886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:28.733162 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.733519 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.733520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.733521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.733573 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:28.733778 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:28.733779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:28.733780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:28.733880 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:28.733934 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:28.734179 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 22:52:28.734250 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:52:28.734484 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:28.726617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:56093)
0620 22:52:28.726791 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:28.726831 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:28.727289 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:28.727524 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:28.727528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:28.728057 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:28.728231 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:28.728235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.728918 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.728919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:28.729689 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:28.729691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:28.730034 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:28.730036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:28.730157 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:28.730565 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:28.730573 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:28.731795 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:28.733230 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:28.733236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:28.733238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:28.733461 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:28.733625 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:28.733625 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:28.733626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:28.733726 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:28.733938 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:28.733943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:28.734067 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 22:52:28.734149 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:52:28.734290 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260620 22:52:28.734838 11027 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35549
I20260620 22:52:28.734913 11027 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:28.735049 11027 heartbeater.cc:511] Master 127.2.90.190:35549 requested a full tablet report, sending...
I20260620 22:52:28.735317 10568 ts_manager.cc:194] Registered new tserver with Master: 0a42e2831ecf46089187d94c6f465b3c (127.2.90.131:34201)
I20260620 22:52:28.735635 10568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:56093
I20260620 22:52:28.744539 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab.
Entry for principal oryx/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab.
WARNING: no policy specified for oryx/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab.
Entry for principal oryx/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab.
WARNING: no policy specified for oryx/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab.
Entry for principal oryx/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab.
I20260620 22:52:28.803205 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 10630
I20260620 22:52:28.806725 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 10764
I20260620 22:52:28.810715 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 10898
I20260620 22:52:28.814194 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 10536
I20260620 22:52:28.818352 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35549
--webserver_interface=127.2.90.190
--webserver_port=33933
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:28.894409 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:28.894609 11036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:28.894629 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:28.895753 11036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:28.895792 11036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:28.895805 11036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:28.895828 11036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:28.895843 11036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:28.897279 11036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=33933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:28.897500 11036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:28.897696 11036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:28.902052 11044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:28.902081 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.902191 11036 server_base.cc:1034] running on GCE node
W20260620 22:52:28.902052 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:28.902441 11036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:28.902694 11036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:28.903827 11036 hybrid_clock.cc:648] HybridClock initialized: now 1781995948903812 us; error 27 us; skew 500 ppm
Jun 20 22:52:28 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:28.906054 11036 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:28.906435 11036 webserver.cc:460] Webserver started at http://127.2.90.190:33933/ using document root <none> and password file <none>
I20260620 22:52:28.906618 11036 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:28.906677 11036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:28.907876 11036 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:28.908406 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:28.908548 11036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:28.908624 11036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "c2024c1f1b8b44318bb3e6dac1841eab"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "d06229c5bcde65b291bc3fad311e1f4c"
server_key_iv: "7deda463b1233fe65f81921c6be6ba02"
server_key_version: "encryptionkey@0"
I20260620 22:52:28.908954 11036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:28.923709 11036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:28.925525 11036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:28.925632 11036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:28.929008 11036 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35549
I20260620 22:52:28.929039 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35549 every 8 connection(s)
I20260620 22:52:28.929339 11036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:28.930608 11104 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:52:28.931156 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Bootstrap starting.
I20260620 22:52:28.932006 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11036
I20260620 22:52:28.932257 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:45655
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:35549
--webserver_port=33757
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:52:28.938699 11104 log.cc:824] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:28.940212 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:52:28.940440 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Bootstrap complete.
I20260620 22:52:28.942452 11104 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.942667 11104 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2024c1f1b8b44318bb3e6dac1841eab, State: Initialized, Role: FOLLOWER
I20260620 22:52:28.942811 11104 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.942884 11104 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:28.942938 11104 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:28.942982 11104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:52:28.943948 11104 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.944041 11104 leader_election.cc:302] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2024c1f1b8b44318bb3e6dac1841eab; no voters:
I20260620 22:52:28.944185 11104 leader_election.cc:288] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:52:28.944226 11108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:52:28.944341 11108 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [term 2 LEADER]: Becoming Leader. State: Replica: c2024c1f1b8b44318bb3e6dac1841eab, State: Running, Role: LEADER
I20260620 22:52:28.944437 11104 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:28.944439 11108 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } }
I20260620 22:52:28.944783 11109 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } } }
I20260620 22:52:28.944787 11108 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2024c1f1b8b44318bb3e6dac1841eab. Latest consensus state: current_term: 2 leader_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2024c1f1b8b44318bb3e6dac1841eab" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35549 } } }
I20260620 22:52:28.944973 11109 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:28.944990 11108 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:28.946110 11124 catalog_manager.cc:1259] Loaded cluster ID: 5ab68ee4c7144eb3b1c6673a53a8a505
I20260620 22:52:28.946153 11124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: loading cluster ID for follower catalog manager: success
I20260620 22:52:28.947268 11124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: acquiring CA information for follower catalog manager: success
I20260620 22:52:28.947592 11124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:52:28.947659 11117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:28.947808 11117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:28.947878 11117 catalog_manager.cc:1259] Loaded cluster ID: 5ab68ee4c7144eb3b1c6673a53a8a505
I20260620 22:52:28.947909 11117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:28.948071 11117 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:28.948144 11117 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c2024c1f1b8b44318bb3e6dac1841eab: Loaded TSK: 0
I20260620 22:52:28.948264 11117 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:52:29.010107 11106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:29.010319 11106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:29.010352 11106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:29.011570 11106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:29.011612 11106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:29.011648 11106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:29.013154 11106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.129.keytab
--principal=oryx/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:45655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=33757
--tserver_master_addrs=127.2.90.190:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:29.013375 11106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:29.013583 11106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:29.018213 11130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:29.018218 11131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:29.018275 11133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:29.018733 11106 server_base.cc:1034] running on GCE node
I20260620 22:52:29.018949 11106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:29.019227 11106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:29.020406 11106 hybrid_clock.cc:648] HybridClock initialized: now 1781995949020385 us; error 33 us; skew 500 ppm
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995949, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:29.023352 11106 init.cc:377] Logged in from keytab as oryx/127.2.90.129@KRBTEST.COM (short username oryx)
I20260620 22:52:29.023904 11106 webserver.cc:460] Webserver started at http://127.2.90.129:33757/ using document root <none> and password file <none>
I20260620 22:52:29.024171 11106 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:29.024255 11106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:29.026005 11106 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:29.026646 11140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.026810 11106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.026906 11106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "cc486ed57d5d49679b7fc2afa24a69bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "226997c020a55a1c48b5f231a5780f31"
server_key_iv: "cb3ead3a8836dc13851375494f15c71d"
server_key_version: "encryptionkey@0"
I20260620 22:52:29.027330 11106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:29.057829 11106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:29.060184 11106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:29.060339 11106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:29.060631 11106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:29.061015 11106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:29.061065 11106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.061100 11106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:29.061122 11106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.067767 11106 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:45655
I20260620 22:52:29.067813 11253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:45655 every 8 connection(s)
I20260620 22:52:29.068193 11106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.076853 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.068861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:41775)
0620 22:52:29.069047 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.069096 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.069720 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.070126 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.070130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.070860 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.071143 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.071150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.072173 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.072174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.073422 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.073425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.073939 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.073940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.074064 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.074518 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.074527 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.076612 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260620 22:52:29.076874 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11106
W20260620 22:52:29.076906 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.068970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:41775 (local address 127.2.90.190:35549)
0620 22:52:29.069133 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.069144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.069172 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:52:29.069898 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.069903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.069949 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.070054 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.071240 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.072078 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.073494 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.073870 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.073945 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.076767 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:41775: BlockingRecv error: recv got EOF from 127.2.90.129:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260620 22:52:29.077162 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:45301
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:35549
--webserver_port=36899
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:29.077442 11254 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35549 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.90.190:35549: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.082073 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.077646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:43593)
0620 22:52:29.077686 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.077696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.077729 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.077836 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.077837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.077955 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.077995 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.077996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.078710 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.078711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.079362 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.079365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.079776 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.079777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.079840 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.080261 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.080266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.082003 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:52:29.082125 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.077686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:43593 (local address 127.2.90.190:35549)
0620 22:52:29.077718 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.077719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.077726 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:29.077773 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.077774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.077784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.077831 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.078053 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.078631 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.079448 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.079706 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.079771 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.082077 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:43593: BlockingRecv error: recv got EOF from 127.2.90.129:43593 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.086001 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.082293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:39739)
0620 22:52:29.082342 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.082353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.082395 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.082521 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.082522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.082636 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.082660 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.082661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.083225 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.083226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.083704 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.083706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.084009 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.084010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.084056 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.084440 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.084443 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.085956 (+ 1513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 22:52:29.086107 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.082347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:39739 (local address 127.2.90.190:35549)
0620 22:52:29.082383 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.082385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.082393 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:29.082457 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.082458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.082468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.082513 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.082718 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.083164 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.083770 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.083956 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.084007 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.086053 (+ 2046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:39739: BlockingRecv error: recv got EOF from 127.2.90.129:39739 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:52:29.086115 11254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:52:29.155378 11258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:29.155577 11258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:29.155606 11258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:29.156828 11258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:29.156875 11258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:29.156910 11258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:29.158475 11258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.130.keytab
--principal=oryx/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:45301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=36899
--tserver_master_addrs=127.2.90.190:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:29.158701 11258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:29.158910 11258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:29.163975 11263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:29.164041 11266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:29.164175 11258 server_base.cc:1034] running on GCE node
W20260620 22:52:29.164266 11264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:29.164518 11258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:29.164793 11258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:29.165961 11258 hybrid_clock.cc:648] HybridClock initialized: now 1781995949165914 us; error 37 us; skew 500 ppm
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995949, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:29.168955 11258 init.cc:377] Logged in from keytab as oryx/127.2.90.130@KRBTEST.COM (short username oryx)
I20260620 22:52:29.169345 11258 webserver.cc:460] Webserver started at http://127.2.90.130:36899/ using document root <none> and password file <none>
I20260620 22:52:29.169557 11258 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:29.169613 11258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:29.170802 11258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:52:29.171276 11273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.171418 11258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.171499 11258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "a6a51e31703b451e8a787ddfe47d81bb"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "ac8f4602837064ffc65e1182ecd4e8a0"
server_key_iv: "a6a05b1117ca57f951ea81b2b9cfa5ec"
server_key_version: "encryptionkey@0"
I20260620 22:52:29.171795 11258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:29.187121 11258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:29.188722 11258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:29.188849 11258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:29.189047 11258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:29.189317 11258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:29.189350 11258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.189376 11258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:29.189393 11258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.194653 11258 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45301
I20260620 22:52:29.194710 11386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45301 every 8 connection(s)
I20260620 22:52:29.195001 11258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:52:29.201339 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.195600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:46193 (local address 127.2.90.190:35549)
0620 22:52:29.195640 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.195641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.195857 (+ 216us) server_negotiation.cc:407] Connection header received
0620 22:52:29.196289 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.196290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.196305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.196351 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.197223 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.197701 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.198612 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.198783 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.198825 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.201284 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:46193: BlockingRecv error: recv got EOF from 127.2.90.130:46193 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:52:29.201367 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.195591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:46193)
0620 22:52:29.195723 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.195756 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.196216 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.196404 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.196407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.196973 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.197153 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.197158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.197763 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.197763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.198552 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.198554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.198833 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.198833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.198919 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.199311 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.199318 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.201186 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260620 22:52:29.201560 11387 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35549 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.90.190:35549: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
I20260620 22:52:29.201941 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11258
I20260620 22:52:29.202180 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:34201
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:35549
--webserver_port=33671
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.205509 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.201753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:44715)
0620 22:52:29.201794 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.201803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.201839 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.201953 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.201954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.202084 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.202123 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.202124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.202737 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.202738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.203219 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.203220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.203523 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.203524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.203581 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.203920 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.203924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.205444 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:52:29.205587 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.201798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:44715 (local address 127.2.90.190:35549)
0620 22:52:29.201851 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.201853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.201860 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:29.201882 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.201883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.201892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.201946 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.202183 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.202672 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.203283 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.203471 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.203519 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.205528 (+ 2009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:44715: BlockingRecv error: recv got EOF from 127.2.90.130:44715 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.209652 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.205851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:42009)
0620 22:52:29.205891 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.205900 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.205940 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.206041 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.206042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.206164 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.206195 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.206196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.206728 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.206728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.207456 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.207458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.207737 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.207738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.207788 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.208095 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.208100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.209599 (+ 1499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 22:52:29.209719 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.205898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:42009 (local address 127.2.90.190:35549)
0620 22:52:29.205935 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.205937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.205944 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:29.205986 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.205987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.205993 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.206035 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.206249 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.206665 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.207523 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.207683 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.207721 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.209674 (+ 1953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:42009: BlockingRecv error: recv got EOF from 127.2.90.130:42009 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 22:52:29.209772 11387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:52:29.280325 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:29.280529 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:29.280551 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:29.281725 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:29.281769 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:29.281798 11390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:29.283210 11390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/krb5kdc/oryx_127.2.90.131.keytab
--principal=oryx/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:34201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=33671
--tserver_master_addrs=127.2.90.190:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:29.283427 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:29.283634 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:29.288224 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:29.288251 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:29.288280 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:29.288362 11390 server_base.cc:1034] running on GCE node
I20260620 22:52:29.288563 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:29.288816 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:29.289953 11390 hybrid_clock.cc:648] HybridClock initialized: now 1781995949289939 us; error 29 us; skew 500 ppm
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995949, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:29.292205 11390 init.cc:377] Logged in from keytab as oryx/127.2.90.131@KRBTEST.COM (short username oryx)
I20260620 22:52:29.292662 11390 webserver.cc:460] Webserver started at http://127.2.90.131:33671/ using document root <none> and password file <none>
I20260620 22:52:29.292922 11390 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:29.292984 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:29.294306 11390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:52:29.294801 11405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.294950 11390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:29.295028 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "0a42e2831ecf46089187d94c6f465b3c"
format_stamp: "Formatted at 2026-06-20 22:52:28 on dist-test-slave-4mtw"
server_key: "bad941078b2407b72999f731c87a7563"
server_key_iv: "b982314c3a597a3f65bce661e7abf485"
server_key_version: "encryptionkey@0"
I20260620 22:52:29.295356 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:29.307905 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:29.309522 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:29.309643 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:29.309832 11390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:29.310101 11390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:29.310132 11390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.310158 11390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:29.310182 11390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:29.315392 11390 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:34201
I20260620 22:52:29.315450 11518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:34201 every 8 connection(s)
I20260620 22:52:29.315739 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:29.316655 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11390
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 22:52:29.323058 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.316324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:35101 (local address 127.2.90.190:35549)
0620 22:52:29.316370 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.316372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.316598 (+ 226us) server_negotiation.cc:407] Connection header received
0620 22:52:29.317233 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.317234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.317245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.317274 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.318353 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.318990 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.320102 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.320279 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.320328 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.323005 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:35101: BlockingRecv error: recv got EOF from 127.2.90.131:35101 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:29.323102 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.316337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:35101)
0620 22:52:29.316483 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.316518 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.317118 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.317445 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.317448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.318020 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.318269 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.318275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.319063 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.319064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.320031 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.320033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.320341 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.320342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.320449 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.320874 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.320883 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.322920 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
W20260620 22:52:29.323278 11519 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35549 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.90.190:35549: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
I20260620 22:52:29.323906 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.317106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.2.90.190:35549)
0620 22:52:29.317213 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.317215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.317223 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:29.317281 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.317281 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.317289 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.317350 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.317882 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.318548 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.319420 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.319719 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.319787 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.320941 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:29.320946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:29.320946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:29.320955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:29.322593 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:29.322903 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:29.322906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:29.322907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:29.322998 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:29.323185 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:29.323186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:29.323186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:29.323465 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:29.323607 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:29.323719 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 22:52:29.323821 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.327054 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.323454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:42405)
0620 22:52:29.323541 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.323551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.323591 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.323693 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.323693 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.323820 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.323870 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.323872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.324558 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.324559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.325101 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.325103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.325375 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.325375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.325422 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.325748 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.325753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.327013 (+ 1260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 22:52:29.327126 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.323496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:42405 (local address 127.2.90.190:35549)
0620 22:52:29.323533 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.323534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.323581 (+ 47us) server_negotiation.cc:407] Connection header received
0620 22:52:29.323634 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.323635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.323643 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.323689 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.323930 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.324499 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.325161 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.325325 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.325370 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.327078 (+ 1708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:42405: BlockingRecv error: recv got EOF from 127.2.90.131:42405 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:29 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:29.330482 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:29.327272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:53547)
0620 22:52:29.327299 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:29.327310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:29.327348 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:29.327444 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:29.327444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:29.327533 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:29.327554 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.327555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.328048 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.328048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.328498 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:29.328499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.328785 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.328786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:29.328819 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.329124 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:29.329129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:29.330443 (+ 1314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 22:52:29.330541 11257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:29.327326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:53547 (local address 127.2.90.190:35549)
0620 22:52:29.327357 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:29.327358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:29.327366 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:29.327385 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:29.327385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:29.327391 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:29.327439 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:29.327596 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.328002 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.328545 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:29.328738 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:29.328781 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:29.330501 (+ 1720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:53547: BlockingRecv error: recv got EOF from 127.2.90.131:53547 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:52:29.330593 11519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 22:52:30 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:30 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:30.090497 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:30.086458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:58577)
0620 22:52:30.086579 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:30.086589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:30.086638 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:30.086755 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:30.086756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:30.086898 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:30.086944 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:30.086946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.087668 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.087669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:30.088199 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:30.088200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.088490 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.088490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:30.088525 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:30.088968 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:30.088972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:30.090437 (+ 1465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260620 22:52:30.090565 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:30.086515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:58577 (local address 127.2.90.190:35549)
0620 22:52:30.086619 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:30.086621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:30.086629 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:30.086685 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:30.086685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:30.086694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:30.086747 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:30.086989 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.087618 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.088254 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.088444 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.088484 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:30.090511 (+ 2027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:58577: BlockingRecv error: recv got EOF from 127.2.90.129:58577 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
Jun 20 22:52:30 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:30 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:30.214143 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:30.210097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:41239)
0620 22:52:30.210220 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:30.210231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:30.210290 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:30.210391 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:30.210391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:30.210547 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:30.210592 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:30.210594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.211208 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.211209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:30.211769 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:30.211771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.212057 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.212058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:30.212100 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:30.212484 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:30.212488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:30.214051 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
W20260620 22:52:30.214211 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:30.210158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:41239 (local address 127.2.90.190:35549)
0620 22:52:30.210222 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:30.210224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:30.210286 (+ 62us) server_negotiation.cc:407] Connection header received
0620 22:52:30.210328 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:30.210328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:30.210338 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:30.210382 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:30.210640 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.211145 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.211827 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.212004 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.212060 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:30.214153 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:41239: BlockingRecv error: recv got EOF from 127.2.90.130:41239 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 22:52:30 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:30 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:30.335186 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:30.330914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:51475)
0620 22:52:30.331057 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:30.331069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:30.331136 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:30.331279 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:30.331280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:30.331434 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:30.331481 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:30.331489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.332161 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.332163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:30.332762 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:30.332764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.333087 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.333088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:30.333137 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:30.333575 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:30.333581 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:30.335116 (+ 1535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
W20260620 22:52:30.335250 11525 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:30.330962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:51475 (local address 127.2.90.190:35549)
0620 22:52:30.331068 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:30.331070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:30.331132 (+ 62us) server_negotiation.cc:407] Connection header received
0620 22:52:30.331186 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:30.331187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:30.331198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:30.331245 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:30.331558 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.332091 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.332832 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:30.333031 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:30.333084 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:30.335197 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:51475: BlockingRecv error: recv got EOF from 127.2.90.131:51475 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 22:52:31 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:31 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:31.097579 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:31.090876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:49315)
0620 22:52:31.091023 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:31.091034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:31.091088 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:31.091243 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:31.091244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:31.091392 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:31.091458 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:31.091460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.092046 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.092046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:31.092571 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:31.092572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.092858 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.092858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:31.092894 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:31.095885 (+ 2991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:31.095890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:31.097523 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
W20260620 22:52:31.097677 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:31.090971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:49315 (local address 127.2.90.190:35549)
0620 22:52:31.091067 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:31.091069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:31.091076 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:31.091135 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:31.091135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:31.091144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:31.091197 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:31.091514 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.091995 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.092621 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.092811 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.092848 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:31.097611 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:49315: BlockingRecv error: recv got EOF from 127.2.90.129:49315 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
Jun 20 22:52:31 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:31 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:31.218557 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:31.214642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:37037)
0620 22:52:31.214773 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:31.214782 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:31.214835 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:31.214955 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:31.214956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:31.215101 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:31.215147 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:31.215149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.215727 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.215728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:31.216247 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:31.216249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.216520 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.216521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:31.216556 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:31.216994 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:31.216998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:31.218500 (+ 1502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260620 22:52:31.218644 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:31.214697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:37037 (local address 127.2.90.190:35549)
0620 22:52:31.214741 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:31.214743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:31.214831 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:52:31.214884 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:31.214885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:31.214895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:31.214941 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:31.215195 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.215674 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.216302 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.216474 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.216516 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:31.218586 (+ 2070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:37037: BlockingRecv error: recv got EOF from 127.2.90.130:37037 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 22:52:31 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:31 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:31.339888 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:31.335625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:53897)
0620 22:52:31.335767 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:31.335779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:31.335848 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:31.335979 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:31.335979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:31.336142 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:31.336187 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:31.336189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.336829 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.336830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:31.337344 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:31.337346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.337783 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.337784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:31.337844 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:31.338266 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:31.338271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:31.339805 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
W20260620 22:52:31.340101 11529 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:31.335733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:53897 (local address 127.2.90.190:35549)
0620 22:52:31.335786 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:31.335787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:31.335829 (+ 42us) server_negotiation.cc:407] Connection header received
0620 22:52:31.335896 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:31.335897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:31.335910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:31.335966 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:31.336255 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.336767 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.337427 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:31.337703 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:31.337765 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:31.339909 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:53897: BlockingRecv error: recv got EOF from 127.2.90.131:53897 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 22:52:32 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:32 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:32.102283 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:32.098030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:58289)
0620 22:52:32.098143 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:32.098154 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:32.098201 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:32.098315 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:32.098316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:32.098483 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:32.098529 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:32.098531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.099148 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.099150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:32.099686 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:32.099688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.099981 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.099981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:32.100017 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:32.100404 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:32.100408 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:32.102213 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
W20260620 22:52:32.102366 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:32.098080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:58289 (local address 127.2.90.190:35549)
0620 22:52:32.098180 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:32.098182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:32.098204 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:52:32.098249 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:32.098249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:32.098259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:32.098302 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:32.098582 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.099091 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.099754 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.099935 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.099975 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:32.102296 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:58289: BlockingRecv error: recv got EOF from 127.2.90.129:58289 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
Jun 20 22:52:32 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:32 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:32.223028 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:32.219002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:38127)
0620 22:52:32.219129 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:32.219139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:32.219190 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:32.219299 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:32.219299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:32.219452 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:32.219502 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:32.219503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.220098 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.220099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:32.220628 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:32.220630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.220967 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.220968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:32.221006 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:32.221391 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:32.221395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:32.222961 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
W20260620 22:52:32.223117 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:32.219045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:38127 (local address 127.2.90.190:35549)
0620 22:52:32.219087 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:32.219089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:32.219179 (+ 90us) server_negotiation.cc:407] Connection header received
0620 22:52:32.219236 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:32.219236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:32.219248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:32.219283 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:32.219555 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.220045 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.220711 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.220904 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.220963 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:32.223055 (+ 2092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:38127: BlockingRecv error: recv got EOF from 127.2.90.130:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 22:52:32 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:32 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:32.344980 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:32.340395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:55831)
0620 22:52:32.340520 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:32.340530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:32.340580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:32.340718 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:32.340719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:32.340870 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:32.340929 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:32.340930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.341514 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.341514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:32.342053 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:32.342055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.342323 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.342324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:32.342360 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:32.342779 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:32.342784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:32.344891 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
W20260620 22:52:32.345077 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:32.340490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:55831 (local address 127.2.90.190:35549)
0620 22:52:32.340555 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:32.340557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:32.340568 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:52:32.340632 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:32.340633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:32.340644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:32.340710 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:32.340980 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.341460 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.342109 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:32.342277 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:32.342318 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:32.345012 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:55831: BlockingRecv error: recv got EOF from 127.2.90.131:55831 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 22:52:33 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:33 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:33.106912 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:33.102693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:33749)
0620 22:52:33.102828 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:33.102839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:33.102900 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:33.103033 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:33.103033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:33.103177 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:33.103244 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:33.103245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.103982 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.103983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:33.104502 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:33.104504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.104855 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.104856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:33.104895 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:33.105283 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:33.105287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:33.106853 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260620 22:52:33.107017 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:33.102745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:33749 (local address 127.2.90.190:35549)
0620 22:52:33.102868 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:33.102871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:33.102884 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:52:33.102958 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:33.102959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:33.102969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:33.103023 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:33.103298 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.103909 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.104571 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.104807 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.104856 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:33.106951 (+ 2095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:33749: BlockingRecv error: recv got EOF from 127.2.90.129:33749 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
Jun 20 22:52:33 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:33 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:33.227542 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:33.223448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:48221)
0620 22:52:33.223557 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:33.223568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:33.223620 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:33.223727 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:33.223727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:33.223873 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:33.223920 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:33.223922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.224509 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.224509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:33.225078 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:33.225080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.225380 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.225381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:33.225424 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:33.225836 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:33.225840 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:33.227482 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
W20260620 22:52:33.227624 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:33.223507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:48221 (local address 127.2.90.190:35549)
0620 22:52:33.223550 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:33.223552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:33.223607 (+ 55us) server_negotiation.cc:407] Connection header received
0620 22:52:33.223659 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:33.223660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:33.223670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:33.223716 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:33.223971 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.224454 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.225146 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.225332 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.225375 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:33.227570 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:48221: BlockingRecv error: recv got EOF from 127.2.90.130:48221 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:52:33 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:33 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:33.350399 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:33.345365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:39813)
0620 22:52:33.345520 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:33.345532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:33.345592 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:33.345716 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:33.345717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:33.345902 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:33.345957 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:33.345958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.346772 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.346774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:33.347440 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:33.347442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.347855 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.347856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:33.347914 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:33.348469 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:33.348474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:33.350321 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
W20260620 22:52:33.350462 11537 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:33.345456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:39813 (local address 127.2.90.190:35549)
0620 22:52:33.345542 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:33.345544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:33.345577 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:52:33.345642 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:33.345643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:33.345655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:33.345704 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:33.346019 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.346685 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.347526 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:33.347787 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:33.347855 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:33.350408 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:39813: BlockingRecv error: recv got EOF from 127.2.90.131:39813 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 22:52:34 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:34 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:34.111330 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:34.107336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:40183)
0620 22:52:34.107469 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:34.107478 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:34.107529 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:34.107643 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:34.107643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:34.107790 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:34.107836 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:34.107837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.108427 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.108427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:34.108972 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:34.108974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.109256 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.109256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:34.109291 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:34.109673 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:34.109677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:34.111269 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
W20260620 22:52:34.111428 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:34.107381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:40183 (local address 127.2.90.190:35549)
0620 22:52:34.107491 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:34.107492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:34.107516 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:52:34.107568 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:34.107568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:34.107577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:34.107635 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:34.107882 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.108369 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.109032 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.109211 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.109253 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:34.111357 (+ 2104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:40183: BlockingRecv error: recv got EOF from 127.2.90.129:40183 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
Jun 20 22:52:34 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:34 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:34.232254 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:34.228045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:56545)
0620 22:52:34.228169 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:34.228180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:34.228247 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:34.228395 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:34.228396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:34.228600 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:34.228648 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:34.228650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.229290 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.229290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:34.229805 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:34.229806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.230110 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.230111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:34.230151 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:34.230525 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:34.230528 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:34.232183 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
W20260620 22:52:34.232335 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:34.228142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:56545 (local address 127.2.90.190:35549)
0620 22:52:34.228184 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:34.228186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:34.228230 (+ 44us) server_negotiation.cc:407] Connection header received
0620 22:52:34.228311 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:34.228312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:34.228324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:34.228380 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:34.228740 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.229230 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.229859 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.230056 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.230094 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:34.232277 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:56545: BlockingRecv error: recv got EOF from 127.2.90.130:56545 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 22:52:34 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:34 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:34.355513 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:34.350849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:47727)
0620 22:52:34.350968 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:34.350978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:34.351042 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:34.351175 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:34.351176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:34.351365 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:34.351421 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:34.351423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.352275 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.352277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:34.353053 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:34.353055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.353362 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.353363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:34.353398 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:34.353792 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:34.353797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:34.355436 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
W20260620 22:52:34.355583 11541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:34.350887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:47727 (local address 127.2.90.190:35549)
0620 22:52:34.350934 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:34.350936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:34.351029 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:52:34.351102 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:34.351103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:34.351116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:34.351166 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:34.351481 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.352183 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.353121 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:34.353306 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:34.353361 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:34.355532 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:47727: BlockingRecv error: recv got EOF from 127.2.90.131:47727 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 22:52:35 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:35 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:35.115715 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:35.111720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:38185)
0620 22:52:35.111835 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:35.111844 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:35.111891 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:35.112018 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:35.112019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:35.112169 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:35.112223 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:35.112236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.112866 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.112866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:35.113394 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:35.113396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.113693 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.113694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:35.113728 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:35.114088 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:35.114092 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:35.115657 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20260620 22:52:35.115804 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:35.111759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:38185 (local address 127.2.90.190:35549)
0620 22:52:35.111869 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:35.111871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:35.111879 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:35.111942 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:35.111943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:35.111952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:35.112008 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:35.112289 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.112813 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.113462 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.113646 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.113691 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:35.115732 (+ 2041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:38185: BlockingRecv error: recv got EOF from 127.2.90.129:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
Jun 20 22:52:35 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:35 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:35.236809 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:35.232673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:39427)
0620 22:52:35.232815 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:35.232826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:35.232881 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:35.232991 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:35.232992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:35.233144 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:35.233188 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:35.233190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.233788 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.233789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:35.234327 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:35.234329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.234615 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.234615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:35.234654 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:35.235179 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:35.235184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:35.236749 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
W20260620 22:52:35.236900 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:35.232735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:39427 (local address 127.2.90.190:35549)
0620 22:52:35.232785 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:35.232787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:35.232873 (+ 86us) server_negotiation.cc:407] Connection header received
0620 22:52:35.232922 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:35.232923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:35.232932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:35.232978 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:35.233234 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.233721 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.234382 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.234564 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.234606 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:35.236835 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:39427: BlockingRecv error: recv got EOF from 127.2.90.130:39427 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 22:52:35 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:35 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:35.359961 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:35.355947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:52051)
0620 22:52:35.356054 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:35.356064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:35.356122 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:35.356227 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:35.356228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:35.356380 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:35.356424 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:35.356425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.357067 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.357068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:35.357580 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:35.357582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.357853 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.357853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:35.357889 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:35.358263 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:35.358268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:35.359882 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":59,"threads_started":1}
W20260620 22:52:35.360045 11545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:35.356012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:52051 (local address 127.2.90.190:35549)
0620 22:52:35.356057 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:35.356059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:35.356109 (+ 50us) server_negotiation.cc:407] Connection header received
0620 22:52:35.356162 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:35.356163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:35.356176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:35.356221 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:35.356473 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.357012 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.357636 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:35.357805 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:35.357845 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:35.359993 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:52051: BlockingRecv error: recv got EOF from 127.2.90.131:52051 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 22:52:36 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:36 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:36.125525 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:36.116165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:51163)
0620 22:52:36.116343 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:36.116353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:36.116418 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:36.116541 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:36.116542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:36.116719 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:36.116772 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:36.116774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.117422 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.117423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:36.117955 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:36.117957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.118242 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.118243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:36.118279 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:36.123868 (+ 5589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:36.123873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:36.125464 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
W20260620 22:52:36.125636 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:36.116268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:51163 (local address 127.2.90.190:35549)
0620 22:52:36.116370 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:36.116372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:36.116402 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:52:36.116470 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:36.116471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:36.116480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:36.116525 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:36.116824 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.117317 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.118016 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.118196 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.118237 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:36.125558 (+ 7321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:51163: BlockingRecv error: recv got EOF from 127.2.90.129:51163 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":44,"threads_started":1}
Jun 20 22:52:36 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:36 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:36.241503 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:36.237307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:48385)
0620 22:52:36.237435 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:36.237445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:36.237502 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:36.237616 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:36.237617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:36.237786 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:36.237831 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:36.237833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.238408 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.238409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:36.238926 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:36.238927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.239204 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.239205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:36.239255 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:36.239705 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:36.239710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:36.241427 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260620 22:52:36.241575 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:36.237389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:48385 (local address 127.2.90.190:35549)
0620 22:52:36.237445 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:36.237447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:36.237498 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:52:36.237545 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:36.237546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:36.237555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:36.237604 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:36.237880 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.238358 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.238989 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.239157 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.239194 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:36.241525 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:48385: BlockingRecv error: recv got EOF from 127.2.90.130:48385 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 22:52:36 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:36 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:36.364398 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:36.360374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:39873)
0620 22:52:36.360539 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:36.360548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:36.360604 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:36.360743 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:36.360744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:36.360893 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:36.360940 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:36.360942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.361515 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.361516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:36.362026 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:36.362027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.362288 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.362288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:36.362330 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:36.362727 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:36.362731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:36.364341 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
W20260620 22:52:36.364473 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:36.360435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:39873 (local address 127.2.90.190:35549)
0620 22:52:36.360491 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:36.360493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:36.360600 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:52:36.360643 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:36.360644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:36.360654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:36.360736 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:36.360989 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.361461 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.362076 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:36.362243 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:36.362284 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:36.364415 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:39873: BlockingRecv error: recv got EOF from 127.2.90.131:39873 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 22:52:37 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:37 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:37.130137 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:37.125955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:37901)
0620 22:52:37.126084 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:37.126094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:37.126141 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:37.126322 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:37.126322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:37.126537 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:37.126587 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:37.126590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.127185 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.127186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:37.127692 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:37.127694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.127968 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.127968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:37.128004 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:37.128384 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:37.128388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:37.130049 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
W20260620 22:52:37.130201 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:37.126066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:37901 (local address 127.2.90.190:35549)
0620 22:52:37.126186 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:37.126188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:37.126196 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:37.126217 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:37.126218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:37.126229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:37.126302 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:37.126650 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.127128 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.127746 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.127921 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.127963 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:37.130141 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:37901: BlockingRecv error: recv got EOF from 127.2.90.129:37901 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
Jun 20 22:52:37 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:37 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:37.246658 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:37.241955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:58189)
0620 22:52:37.242088 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:37.242097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:37.242159 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:37.242270 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:37.242270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:37.242432 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:37.242478 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:37.242480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.243367 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.243369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:37.244202 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:37.244206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.244575 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.244576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:37.244617 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:37.245064 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:37.245069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:37.246599 (+ 1530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260620 22:52:37.246740 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:37.242005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:58189 (local address 127.2.90.190:35549)
0620 22:52:37.242069 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:37.242071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:37.242152 (+ 81us) server_negotiation.cc:407] Connection header received
0620 22:52:37.242202 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:37.242203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:37.242213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:37.242258 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:37.242527 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.243265 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.244276 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.244517 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.244571 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:37.246682 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:58189: BlockingRecv error: recv got EOF from 127.2.90.130:58189 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 22:52:37 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:37 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:37.368997 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:37.364847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:45049)
0620 22:52:37.364964 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:37.364975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:37.365044 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:37.365166 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:37.365166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:37.365337 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:37.365382 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:37.365383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.366047 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.366049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:37.366568 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:37.366570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.366897 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.366898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:37.366952 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:37.367351 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:37.367355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:37.368937 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
W20260620 22:52:37.369089 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:37.364895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:45049 (local address 127.2.90.190:35549)
0620 22:52:37.364961 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:37.364963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:37.365025 (+ 62us) server_negotiation.cc:407] Connection header received
0620 22:52:37.365093 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:37.365094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:37.365107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:37.365159 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:37.365454 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.365973 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.366641 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:37.366840 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:37.366894 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:37.369028 (+ 2134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:45049: BlockingRecv error: recv got EOF from 127.2.90.131:45049 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 22:52:38 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:38 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:38.134757 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:38.130569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:42651)
0620 22:52:38.130707 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:38.130721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:38.130778 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:38.130885 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:38.130886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:38.131049 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:38.131098 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:38.131100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.131750 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.131750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:38.132291 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:38.132293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.132599 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.132600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:38.132636 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:38.133042 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:38.133046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:38.134698 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
W20260620 22:52:38.134867 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:38.130643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:42651 (local address 127.2.90.190:35549)
0620 22:52:38.130766 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:38.130768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:38.130775 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:38.130819 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:38.130819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:38.130829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:38.130881 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:38.131147 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.131676 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.132347 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.132550 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.132597 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:38.134791 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:42651: BlockingRecv error: recv got EOF from 127.2.90.129:42651 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
Jun 20 22:52:38 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:38 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:38.251050 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:38.247044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:44569)
0620 22:52:38.247193 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:38.247204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:38.247256 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:38.247367 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:38.247367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:38.247530 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:38.247574 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:38.247576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.248158 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.248158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:38.248721 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:38.248723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.249011 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.249012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:38.249050 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:38.249454 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:38.249458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:38.250992 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260620 22:52:38.251139 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:38.247111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:44569 (local address 127.2.90.190:35549)
0620 22:52:38.247156 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:38.247157 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:38.247252 (+ 95us) server_negotiation.cc:407] Connection header received
0620 22:52:38.247314 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:38.247315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:38.247326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:38.247361 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:38.247620 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.248104 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.248776 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.248965 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.249007 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:38.251082 (+ 2075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:44569: BlockingRecv error: recv got EOF from 127.2.90.130:44569 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 22:52:38 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:38 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:38.373759 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:38.369406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:38423)
0620 22:52:38.369521 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:38.369534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:38.369602 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:38.369729 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:38.369730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:38.369884 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:38.369941 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:38.369942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.370564 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.370565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:38.371086 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:38.371089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.371397 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.371398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:38.371452 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:38.371865 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:38.371870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:38.373701 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
W20260620 22:52:38.373842 11557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:38.369470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:38423 (local address 127.2.90.190:35549)
0620 22:52:38.369528 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:38.369529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:38.369591 (+ 62us) server_negotiation.cc:407] Connection header received
0620 22:52:38.369666 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:38.369667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:38.369680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:38.369719 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:38.370007 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.370494 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.371160 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:38.371341 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:38.371394 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:38.373782 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:38423: BlockingRecv error: recv got EOF from 127.2.90.131:38423 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 22:52:39 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:39 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:39.139346 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:39.135144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:39921)
0620 22:52:39.135262 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:39.135273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:39.135318 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:39.135464 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:39.135465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:39.135615 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:39.135663 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:39.135665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.136244 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.136244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:39.136939 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:39.136942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.137243 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.137244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:39.137278 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:39.137661 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:39.137665 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:39.139288 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260620 22:52:39.139451 11561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:39.135205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:39921 (local address 127.2.90.190:35549)
0620 22:52:39.135311 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:39.135313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:39.135320 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:52:39.135359 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:39.135360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:39.135369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:39.135455 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:39.135712 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.136194 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.136999 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.137196 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.137241 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:39.139368 (+ 2127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:39921: BlockingRecv error: recv got EOF from 127.2.90.129:39921 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
Jun 20 22:52:39 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:39 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:39.255865 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:39.251475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:39603)
0620 22:52:39.251621 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:39.251633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:39.251695 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:39.251819 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:39.251820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:39.251987 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:39.252029 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:39.252031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.252653 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.252655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:39.253406 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:39.253409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.253723 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.253724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:39.253771 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:39.254161 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:39.254165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:39.255806 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
W20260620 22:52:39.255949 11561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:39.251551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:39603 (local address 127.2.90.190:35549)
0620 22:52:39.251607 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:39.251609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:39.251686 (+ 77us) server_negotiation.cc:407] Connection header received
0620 22:52:39.251748 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:39.251749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:39.251761 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:39.251806 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:39.252097 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.252586 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.253485 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.253668 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.253719 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:39.255890 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:39603: BlockingRecv error: recv got EOF from 127.2.90.130:39603 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 22:52:39 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:39 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:39.378050 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:39.374171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:43135)
0620 22:52:39.374296 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:39.374307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:39.374364 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:39.374504 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:39.374505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:39.374648 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:39.374693 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:39.374694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.375271 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.375271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:39.375826 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:39.375828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.376117 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.376118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:39.376153 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:39.376523 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:39.376528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:39.377994 (+ 1466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
W20260620 22:52:39.378129 11561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:39.374296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:43135 (local address 127.2.90.190:35549)
0620 22:52:39.374372 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:39.374373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:39.374383 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:39.374414 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:39.374415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:39.374439 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:39.374485 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:39.374748 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.375216 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.375884 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:39.376067 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:39.376114 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:39.378072 (+ 1958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:43135: BlockingRecv error: recv got EOF from 127.2.90.131:43135 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:52:40 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:40 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:40.143888 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:40.139770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:59739)
0620 22:52:40.139881 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:40.139891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:40.139942 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:40.140075 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:40.140076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:40.140233 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:40.140281 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:40.140283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.140898 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.140899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:40.141463 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:40.141465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.141756 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.141757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:40.141798 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:40.142230 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:40.142235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:40.143821 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260620 22:52:40.143985 11565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:40.139860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:59739 (local address 127.2.90.190:35549)
0620 22:52:40.139969 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:40.139971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:40.139979 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:40.139998 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:40.139999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:40.140008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:40.140063 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:40.140335 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.140836 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.141518 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.141697 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.141737 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:40.143902 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:59739: BlockingRecv error: recv got EOF from 127.2.90.129:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":46,"threads_started":1}
Jun 20 22:52:40 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:40 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:40.260397 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:40.256289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:39253)
0620 22:52:40.256411 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:40.256421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:40.256477 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:40.256596 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:40.256597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:40.256781 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:40.256826 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:40.256827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.257416 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.257416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:40.257951 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:40.257952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.258219 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.258219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:40.258254 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:40.258688 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:40.258692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:40.260305 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260620 22:52:40.260468 11565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:40.256351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:39253 (local address 127.2.90.190:35549)
0620 22:52:40.256408 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:40.256409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:40.256472 (+ 63us) server_negotiation.cc:407] Connection header received
0620 22:52:40.256535 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:40.256535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:40.256544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:40.256581 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:40.256884 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.257354 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.258004 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.258175 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.258215 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:40.260402 (+ 2187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:39253: BlockingRecv error: recv got EOF from 127.2.90.130:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 22:52:40 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:40 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:40.382530 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:40.378457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:60055)
0620 22:52:40.378596 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:40.378606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:40.378661 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:40.378775 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:40.378775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:40.378949 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:40.378988 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:40.378989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.379597 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.379598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:40.380122 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:40.380123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.380436 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.380437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:40.380491 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:40.380908 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:40.380912 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:40.382439 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260620 22:52:40.382575 11565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:40.378484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:60055 (local address 127.2.90.190:35549)
0620 22:52:40.378522 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:40.378523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:40.378655 (+ 132us) server_negotiation.cc:407] Connection header received
0620 22:52:40.378714 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:40.378714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:40.378727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:40.378771 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:40.379049 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.379529 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.380190 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:40.380373 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:40.380423 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:40.382519 (+ 2096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:60055: BlockingRecv error: recv got EOF from 127.2.90.131:60055 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Jun 20 22:52:41 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:41 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:41.151432 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:41.144262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:49683)
0620 22:52:41.144375 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:41.144384 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:41.144436 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:41.144573 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:41.144574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:41.144760 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:41.144805 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:41.144807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.145473 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.145474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:41.146066 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:41.146068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.146421 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.146422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:41.146467 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:41.149725 (+ 3258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:41.149729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:41.151356 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260620 22:52:41.151501 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:41.144317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:49683 (local address 127.2.90.190:35549)
0620 22:52:41.144408 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:41.144410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:41.144423 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:52:41.144479 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:41.144479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:41.144489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:41.144537 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:41.144856 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.145387 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.146155 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.146351 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.146400 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:41.151442 (+ 5042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:49683: BlockingRecv error: recv got EOF from 127.2.90.129:49683 (error 108)
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
Jun 20 22:52:41 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:41 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:41.265303 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:41.260945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:50705)
0620 22:52:41.261111 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:41.261122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:41.261186 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:41.261329 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:41.261330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:41.261502 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:41.261548 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:41.261550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.262174 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.262175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:41.262692 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:41.262693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.263016 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.263017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:41.263070 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:41.263482 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:41.263485 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:41.265233 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
W20260620 22:52:41.265372 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:41.261040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:50705 (local address 127.2.90.190:35549)
0620 22:52:41.261116 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:41.261118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:41.261186 (+ 68us) server_negotiation.cc:407] Connection header received
0620 22:52:41.261252 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:41.261252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:41.261263 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:41.261309 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:41.261622 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.262104 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.262768 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.262957 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.263011 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:41.265320 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:50705: BlockingRecv error: recv got EOF from 127.2.90.130:50705 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 22:52:41 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:41 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:41.387027 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:41.382945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:45241)
0620 22:52:41.383058 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:41.383068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:41.383120 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:41.383231 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:41.383232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:41.383381 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:41.383436 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:41.383438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.384053 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.384054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:41.384571 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:41.384573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.384921 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.384922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:41.384975 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:41.385384 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:41.385388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:41.386956 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
W20260620 22:52:41.387089 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:41.382991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:45241 (local address 127.2.90.190:35549)
0620 22:52:41.383041 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:41.383043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:41.383115 (+ 72us) server_negotiation.cc:407] Connection header received
0620 22:52:41.383163 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:41.383164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:41.383174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:41.383221 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:41.383505 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.383986 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.384638 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:41.384864 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:41.384918 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:41.387030 (+ 2112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:45241: BlockingRecv error: recv got EOF from 127.2.90.131:45241 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 22:52:42 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:42 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:42.156069 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:42.151868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:60097)
0620 22:52:42.151993 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:42.152002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:42.152056 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:42.152188 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:42.152189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:42.152356 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:42.152403 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:42.152405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.153048 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.153049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:42.153587 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:42.153588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.153876 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.153877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:42.153914 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:42.154310 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:42.154314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:42.155998 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20260620 22:52:42.156152 11573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:42.151966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:60097 (local address 127.2.90.190:35549)
0620 22:52:42.152084 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:42.152086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:42.152094 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:42.152114 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:42.152114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:42.152124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:42.152179 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:42.152466 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.152997 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.153645 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.153827 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.153868 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:42.156087 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:60097: BlockingRecv error: recv got EOF from 127.2.90.129:60097 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
Jun 20 22:52:42 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:42 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:42.269781 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:42.265762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:37747)
0620 22:52:42.265889 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:42.265898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:42.265945 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:42.266081 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:42.266081 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:42.266234 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:42.266278 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:42.266280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.266895 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.266896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:42.267432 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:42.267434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.267715 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.267715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:42.267752 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:42.268137 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:42.268141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:42.269712 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260620 22:52:42.269862 11573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:42.265920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:37747 (local address 127.2.90.190:35549)
0620 22:52:42.265972 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:42.265974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:42.265982 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:42.266006 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:42.266007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:42.266016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:42.266065 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:42.266343 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.266835 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.267490 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.267667 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.267711 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:42.269804 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:37747: BlockingRecv error: recv got EOF from 127.2.90.130:37747 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 22:52:42 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:42 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:42.391719 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:42.387476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:37183)
0620 22:52:42.387601 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:42.387611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:42.387674 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:42.387782 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:42.387783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:42.387938 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:42.387984 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:42.387986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.388572 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.388573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:42.389149 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:42.389151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.389431 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.389431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:42.389467 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:42.389961 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:42.389968 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:42.391633 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
W20260620 22:52:42.391774 11573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:42.387541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37183 (local address 127.2.90.190:35549)
0620 22:52:42.387597 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:42.387599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:42.387666 (+ 67us) server_negotiation.cc:407] Connection header received
0620 22:52:42.387716 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:42.387717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:42.387727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:42.387773 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:42.388031 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.388520 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.389203 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:42.389378 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:42.389422 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:42.391722 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:37183: BlockingRecv error: recv got EOF from 127.2.90.131:37183 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 22:52:43 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:43 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:43.160720 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:43.156492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:57095)
0620 22:52:43.156619 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:43.156630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:43.156697 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:43.156826 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:43.156826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:43.156999 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:43.157047 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:43.157049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.157649 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.157650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:43.158198 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:43.158199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.158536 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.158537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:43.158591 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:43.159023 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:43.159027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:43.160626 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
W20260620 22:52:43.160835 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:43.156557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:57095 (local address 127.2.90.190:35549)
0620 22:52:43.156708 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:43.156709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:43.156735 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:52:43.156757 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:43.156758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:43.156768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:43.156816 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:43.157101 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.157593 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.158255 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.158442 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.158484 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:43.160748 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:57095: BlockingRecv error: recv got EOF from 127.2.90.129:57095 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
Jun 20 22:52:43 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:43 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:43.274297 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:43.270237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:46801)
0620 22:52:43.270354 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:43.270364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:43.270420 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:43.270527 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:43.270527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:43.270686 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:43.270730 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:43.270732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.271303 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.271303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:43.271823 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:43.271824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.272095 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.272095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:43.272133 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:43.272527 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:43.272531 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:43.274229 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260620 22:52:43.274366 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:43.270335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:46801 (local address 127.2.90.190:35549)
0620 22:52:43.270381 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:43.270383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:43.270408 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:52:43.270463 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:43.270463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:43.270473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:43.270517 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:43.270780 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.271251 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.271873 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.272046 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.272086 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:43.274310 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:46801: BlockingRecv error: recv got EOF from 127.2.90.130:46801 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 22:52:43 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:43 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.131@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:43.396216 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:43.392158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.131:33137)
0620 22:52:43.392282 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:43.392293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:43.392357 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:43.392482 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:43.392483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:43.392647 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:43.392716 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:43.392719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.393331 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.393332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:43.393856 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:43.393858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.394185 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.394186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:43.394241 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:43.394636 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:43.394640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:43.396158 (+ 1518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260620 22:52:43.396306 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:43.392223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:33137 (local address 127.2.90.190:35549)
0620 22:52:43.392293 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:43.392295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:43.392346 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:52:43.392410 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:43.392411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:43.392430 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:43.392477 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:43.392784 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.393264 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.393926 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:43.394128 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:43.394181 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:43.396246 (+ 2065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.131:33137: BlockingRecv error: recv got EOF from 127.2.90.131:33137 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 22:52:44 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:44 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.129@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:44.165524 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.161186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.129:42441)
0620 22:52:44.161296 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:44.161309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:44.161361 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:44.161483 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:44.161483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:44.161635 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:44.161682 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.161683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.162290 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.162291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.162868 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.162871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.163295 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.163296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.163355 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.163751 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:44.163755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:44.165466 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
W20260620 22:52:44.165619 11581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:44.161241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:42441 (local address 127.2.90.190:35549)
0620 22:52:44.161374 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.161376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.161385 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:52:44.161407 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.161407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.161417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.161473 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.161738 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.162237 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.162945 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.163224 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.163285 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.165550 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.129:42441: BlockingRecv error: recv got EOF from 127.2.90.129:42441 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
Jun 20 22:52:44 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 22:52:44 dist-test-slave-4mtw krb5kdc[10523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.90.130@KRBTEST.COM for oryx/127.2.90.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 22:52:44.278923 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.274712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35549 (local address 127.2.90.130:57505)
0620 22:52:44.274822 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:44.274833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:44.274887 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:44.275025 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:44.275026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:44.275178 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:44.275224 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.275226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.275894 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.275895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.276459 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.276462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.276811 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.276812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.276856 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.277260 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:44.277264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:44.278861 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.90.190:35549: Server oryx/127.2.90.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
W20260620 22:52:44.279009 11581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:44.274771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:57505 (local address 127.2.90.190:35549)
0620 22:52:44.274824 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.274826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.274882 (+ 56us) server_negotiation.cc:407] Connection header received
0620 22:52:44.274943 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.274943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.274953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.275015 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.275322 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.275820 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.276522 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.276754 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.276802 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.278950 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.90.130:57505: BlockingRecv error: recv got EOF from 127.2.90.130:57505 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:52:44.323674 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11106
I20260620 22:52:44.327512 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11258
I20260620 22:52:44.331056 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11390
I20260620 22:52:44.334401 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11036
2026-06-20T22:52:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17208 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T22:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:44Z Disabled control of system clock
I20260620 22:52:44.359102 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37077
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 22:52:44.435998 11586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:44.436182 11586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:44.436200 11586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:44.437369 11586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:44.437407 11586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:44.437418 11586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:44.437429 11586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:44.437453 11586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:44.438742 11586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:37077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:44.438966 11586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:44.439219 11586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:44.443701 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.443698 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.443819 11586 server_base.cc:1034] running on GCE node
W20260620 22:52:44.443737 11591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.444124 11586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:44.444396 11586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:44.445536 11586 hybrid_clock.cc:648] HybridClock initialized: now 1781995964445521 us; error 27 us; skew 500 ppm
I20260620 22:52:44.446623 11586 webserver.cc:460] Webserver started at http://127.2.90.190:41485/ using document root <none> and password file <none>
I20260620 22:52:44.446820 11586 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:44.446866 11586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:44.446980 11586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:44.447978 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "2f6ea120580b4f398989f33cf9df63a0"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "73f3fcd6560a62f7fbdbf5a0d4cc2a36"
server_key_iv: "7b745f2ce0fb5627535e11e212b00446"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.448288 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "2f6ea120580b4f398989f33cf9df63a0"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "73f3fcd6560a62f7fbdbf5a0d4cc2a36"
server_key_iv: "7b745f2ce0fb5627535e11e212b00446"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.449481 11586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:44.450136 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.450273 11586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:44.450351 11586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "2f6ea120580b4f398989f33cf9df63a0"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "73f3fcd6560a62f7fbdbf5a0d4cc2a36"
server_key_iv: "7b745f2ce0fb5627535e11e212b00446"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.450407 11586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:44.462355 11586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:44.462597 11586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:44.462711 11586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:44.466174 11586 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:37077
I20260620 22:52:44.466202 11652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:37077 every 8 connection(s)
I20260620 22:52:44.466502 11586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:44.467116 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:44.469682 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0: Bootstrap starting.
I20260620 22:52:44.470217 11653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:44.470477 11653 log.cc:824] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:44.471071 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0: No bootstrap required, opened a new log
I20260620 22:52:44.472615 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11586
I20260620 22:52:44.472713 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:44.472620 11653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6ea120580b4f398989f33cf9df63a0" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37077 } }
I20260620 22:52:44.472813 11653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:44.472859 11653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6ea120580b4f398989f33cf9df63a0, State: Initialized, Role: FOLLOWER
I20260620 22:52:44.472905 2410 external_mini_cluster.cc:1428] Setting key 59d9d6fc7c2048ddd1f1df8afee6001c
I20260620 22:52:44.473008 11653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6ea120580b4f398989f33cf9df63a0" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37077 } }
I20260620 22:52:44.473083 11653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:44.473114 11653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:44.473160 11653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:44.473770 11653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6ea120580b4f398989f33cf9df63a0" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37077 } }
I20260620 22:52:44.473865 11653 leader_election.cc:302] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f6ea120580b4f398989f33cf9df63a0; no voters:
I20260620 22:52:44.474013 11653 leader_election.cc:288] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:44.474054 11658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:44.474208 11658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6ea120580b4f398989f33cf9df63a0, State: Running, Role: LEADER
I20260620 22:52:44.474299 11653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:44.474320 11658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6ea120580b4f398989f33cf9df63a0" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37077 } }
I20260620 22:52:44.474645 11658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6ea120580b4f398989f33cf9df63a0. Latest consensus state: current_term: 1 leader_uuid: "2f6ea120580b4f398989f33cf9df63a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6ea120580b4f398989f33cf9df63a0" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37077 } } }
I20260620 22:52:44.474730 11658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:44.474658 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f6ea120580b4f398989f33cf9df63a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6ea120580b4f398989f33cf9df63a0" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 37077 } } }
I20260620 22:52:44.474858 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:44.474958 11664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:44.475476 11664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:44.477065 11664 catalog_manager.cc:1347] Generated new cluster ID: 783399d8d2fe4fda8a6a6330b8b9257e
I20260620 22:52:44.477123 11664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:44.479028 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.473455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.2.90.190:37077)
0620 22:52:44.473649 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.473656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.473684 (+ 28us) server_negotiation.cc:407] Connection header received
0620 22:52:44.473779 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.473784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.473842 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.473964 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.474229 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.475997 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.477003 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.477563 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.477670 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.478372 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:44.478376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:44.478376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:44.478569 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:44.478614 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:44.478628 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:44.478681 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:44.478772 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:52:44.478841 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260620 22:52:44.486649 11664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:44.487088 11664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:44.497040 11664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f6ea120580b4f398989f33cf9df63a0: Generated new TSK 0
I20260620 22:52:44.497174 11664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:44.498417 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37077
--builtin_ntp_servers=127.2.90.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 22:52:44.596531 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:44.596765 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:44.596789 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:44.597939 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:44.597975 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:44.597998 11677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:44.599274 11677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:44.599491 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:44.599766 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:44.605321 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.605324 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.605360 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.605412 11677 server_base.cc:1034] running on GCE node
I20260620 22:52:44.605717 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:44.605967 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:44.607118 11677 hybrid_clock.cc:648] HybridClock initialized: now 1781995964607108 us; error 26 us; skew 500 ppm
I20260620 22:52:44.608304 11677 webserver.cc:460] Webserver started at http://127.2.90.129:37401/ using document root <none> and password file <none>
I20260620 22:52:44.608553 11677 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:44.608598 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:44.608754 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:44.609786 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "7995b59d660a4c2ab8b0119ec59cc9bd"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "f1e3320323c1d9acc7984cd43e98ed6a"
server_key_iv: "f08f4307fc91238e03efe718d77822ce"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.610108 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "7995b59d660a4c2ab8b0119ec59cc9bd"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "f1e3320323c1d9acc7984cd43e98ed6a"
server_key_iv: "f08f4307fc91238e03efe718d77822ce"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.611289 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:44.611964 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.612108 11677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:44.612177 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "7995b59d660a4c2ab8b0119ec59cc9bd"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "f1e3320323c1d9acc7984cd43e98ed6a"
server_key_iv: "f08f4307fc91238e03efe718d77822ce"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.612258 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:44.629758 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:44.630048 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:44.630172 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:44.630378 11677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:44.630692 11677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:44.630726 11677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.630753 11677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:44.630771 11677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.636060 11677 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:42725
I20260620 22:52:44.636114 11804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:42725 every 8 connection(s)
I20260620 22:52:44.636417 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:44.641026 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.637113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:40601 (local address 127.2.90.190:37077)
0620 22:52:44.637190 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.637193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.637379 (+ 186us) server_negotiation.cc:407] Connection header received
0620 22:52:44.637859 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.637860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.637871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.637930 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.638525 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.639011 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.639909 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.640092 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.640134 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.640822 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:44.640823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:44.640824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:44.640831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:44.640846 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:44.640850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:44.640891 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:44.640940 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:52:44.640968 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:52:44.641193 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.637089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37077 (local address 127.2.90.129:40601)
0620 22:52:44.637269 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:44.637302 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:44.637780 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:44.637989 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:44.637992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:44.638001 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:44.638465 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.638470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.639070 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.639071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.639852 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.639855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.640146 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.640147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.640245 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.640702 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:44.640702 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:44.640729 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:44.640752 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:44.640890 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:44.640892 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:44.640895 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:44.640935 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:52:44.641006 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260620 22:52:44.641619 11805 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37077
I20260620 22:52:44.641700 11805 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:44.641866 11805 heartbeater.cc:511] Master 127.2.90.190:37077 requested a full tablet report, sending...
I20260620 22:52:44.642311 11617 ts_manager.cc:194] Registered new tserver with Master: 7995b59d660a4c2ab8b0119ec59cc9bd (127.2.90.129:42725)
I20260620 22:52:44.642889 11617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.129:40601
I20260620 22:52:44.643576 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11677
I20260620 22:52:44.643643 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:44.643826 2410 external_mini_cluster.cc:1428] Setting key dbc9182909ebf386edb266fe14b2c740
I20260620 22:52:44.644579 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37077
--builtin_ntp_servers=127.2.90.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 22:52:44.722082 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:44.722263 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:44.722282 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:44.723369 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:44.723405 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:44.723429 11808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:44.724671 11808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:44.724901 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:44.725152 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:44.729637 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.729750 11808 server_base.cc:1034] running on GCE node
W20260620 22:52:44.729647 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.729664 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.730057 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:44.730273 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:44.731405 11808 hybrid_clock.cc:648] HybridClock initialized: now 1781995964731386 us; error 37 us; skew 500 ppm
I20260620 22:52:44.732533 11808 webserver.cc:460] Webserver started at http://127.2.90.130:44601/ using document root <none> and password file <none>
I20260620 22:52:44.732756 11808 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:44.732800 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:44.732914 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:44.733881 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "ac09fff9e7ff44308b7d9d5f88dc9291"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "921698c217f02fef952bcb637a9b5689"
server_key_iv: "421d307c2f56d43ffa5fdedc0b9f3d0f"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.734198 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "ac09fff9e7ff44308b7d9d5f88dc9291"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "921698c217f02fef952bcb637a9b5689"
server_key_iv: "421d307c2f56d43ffa5fdedc0b9f3d0f"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.735340 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:52:44.735932 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.736061 11808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:44.736140 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "ac09fff9e7ff44308b7d9d5f88dc9291"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "921698c217f02fef952bcb637a9b5689"
server_key_iv: "421d307c2f56d43ffa5fdedc0b9f3d0f"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.736197 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:44.750727 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:44.750980 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:44.751099 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:44.751287 11808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:44.751577 11808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:44.751608 11808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.751636 11808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:44.751653 11808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.756892 11808 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:35371
I20260620 22:52:44.756955 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:35371 every 8 connection(s)
I20260620 22:52:44.757223 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:44.758428 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11808
I20260620 22:52:44.758503 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:44.758663 2410 external_mini_cluster.cc:1428] Setting key b83cb2e83dda05c5bf01e14950b17ca3
I20260620 22:52:44.759545 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37077
--builtin_ntp_servers=127.2.90.148:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 22:52:44.762460 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.757825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:43147 (local address 127.2.90.190:37077)
0620 22:52:44.757879 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.757881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.758086 (+ 205us) server_negotiation.cc:407] Connection header received
0620 22:52:44.758601 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.758602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.758614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.758662 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.759435 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.759972 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.761200 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.761393 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.761466 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.762212 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:44.762213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:44.762214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:44.762218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:44.762233 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:44.762238 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:44.762296 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:44.762345 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 22:52:44.762372 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:52:44.762642 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.757807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37077 (local address 127.2.90.130:43147)
0620 22:52:44.757978 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:44.758016 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:44.758463 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:44.758728 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:44.758732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:44.758742 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:44.759349 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.759355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.760050 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.760051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.761125 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.761128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.761472 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.761473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.761597 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.762087 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:44.762088 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:44.762115 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:44.762136 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:44.762298 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:44.762300 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:44.762303 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:44.762344 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:52:44.762425 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260620 22:52:44.763064 11936 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37077
I20260620 22:52:44.763150 11936 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:44.763306 11936 heartbeater.cc:511] Master 127.2.90.190:37077 requested a full tablet report, sending...
I20260620 22:52:44.763624 11617 ts_manager.cc:194] Registered new tserver with Master: ac09fff9e7ff44308b7d9d5f88dc9291 (127.2.90.130:35371)
I20260620 22:52:44.763970 11617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.130:43147
W20260620 22:52:44.835657 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:44.835841 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:44.835860 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:44.837002 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:44.837040 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:44.837065 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:44.838375 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:44.838573 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:44.838812 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:44.843295 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.843356 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.843506 11939 server_base.cc:1034] running on GCE node
W20260620 22:52:44.843356 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.843748 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:44.843964 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:44.845109 11939 hybrid_clock.cc:648] HybridClock initialized: now 1781995964845095 us; error 29 us; skew 500 ppm
I20260620 22:52:44.846242 11939 webserver.cc:460] Webserver started at http://127.2.90.131:32955/ using document root <none> and password file <none>
I20260620 22:52:44.846467 11939 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:44.846518 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:44.846643 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:44.847656 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "5dfaf9c7c3a24b3a9045be0874033a97"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "d233235f37f70ba536de19a26c8e2985"
server_key_iv: "8fcd3debcaf6801cc8f9333fc68042bd"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.847967 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "5dfaf9c7c3a24b3a9045be0874033a97"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "d233235f37f70ba536de19a26c8e2985"
server_key_iv: "8fcd3debcaf6801cc8f9333fc68042bd"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.849170 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:44.849787 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.849913 11939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:44.849992 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "5dfaf9c7c3a24b3a9045be0874033a97"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "d233235f37f70ba536de19a26c8e2985"
server_key_iv: "8fcd3debcaf6801cc8f9333fc68042bd"
server_key_version: "encryptionkey@0"
I20260620 22:52:44.850051 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:44.857813 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:44.858045 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:44.858156 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:44.858330 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:44.858605 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:44.858636 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.858664 11939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:44.858687 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:44.863808 11939 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:45725
I20260620 22:52:44.863843 12066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:45725 every 8 connection(s)
I20260620 22:52:44.864122 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:44.868569 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.864760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:51117 (local address 127.2.90.190:37077)
0620 22:52:44.864812 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.864815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.865059 (+ 244us) server_negotiation.cc:407] Connection header received
0620 22:52:44.865512 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.865513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.865523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.865571 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.866156 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.866631 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.867499 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.867669 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.867710 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.868391 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:44.868392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:44.868392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:44.868396 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:44.868405 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:44.868408 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:44.868442 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:44.868497 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 22:52:44.868514 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:52:44.868767 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:44.864776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:37077 (local address 127.2.90.131:51117)
0620 22:52:44.864953 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:44.864984 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:44.865441 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:44.865624 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:44.865627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:44.865635 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:44.866097 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.866101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.866687 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.866688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.867443 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:44.867445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:44.867721 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:44.867722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:44.867821 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:44.868285 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:44.868285 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:44.868313 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:44.868334 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:44.868449 (+ 115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:44.868452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:44.868454 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:44.868494 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:52:44.868569 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260620 22:52:44.869230 12067 heartbeater.cc:347] Connected to a master server at 127.2.90.190:37077
I20260620 22:52:44.869302 12067 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:44.869438 12067 heartbeater.cc:511] Master 127.2.90.190:37077 requested a full tablet report, sending...
I20260620 22:52:44.869719 11617 ts_manager.cc:194] Registered new tserver with Master: 5dfaf9c7c3a24b3a9045be0874033a97 (127.2.90.131:45725)
I20260620 22:52:44.870054 11617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.131:51117
I20260620 22:52:44.873735 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 11939
I20260620 22:52:44.873804 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:44.873970 2410 external_mini_cluster.cc:1428] Setting key f81909751ddd218f1cf4338846a403af
I20260620 22:52:44.874723 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:52:44.876338 11657 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:44.875902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.2.90.190:37077)
0620 22:52:44.875944 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:44.875945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:44.876056 (+ 111us) server_negotiation.cc:407] Connection header received
0620 22:52:44.876114 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:44.876115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:44.876126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:44.876169 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:44.876281 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58296: BlockingRecv error: recv got EOF from 127.0.0.1:58296 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:52:44.876399 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11677
I20260620 22:52:44.881068 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11808
I20260620 22:52:44.884483 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11939
I20260620 22:52:44.887904 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 11586
2026-06-20T22:52:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (552 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T22:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:44Z Disabled control of system clock
I20260620 22:52:44.911603 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42555
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:52:44.990803 12078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:44.990996 12078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:44.991016 12078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:44.992069 12078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:44.992100 12078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:44.992112 12078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:44.992123 12078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:44.992137 12078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:44.993414 12078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:42555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:44.993590 12078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:44.993806 12078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:52:44.998150 12078 server_base.cc:1034] running on GCE node
W20260620 22:52:44.998116 12083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.998088 12084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:44.998092 12086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:44.998529 12078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:44.998773 12078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:44.999900 12078 hybrid_clock.cc:648] HybridClock initialized: now 1781995964999884 us; error 32 us; skew 500 ppm
I20260620 22:52:45.001042 12078 webserver.cc:460] Webserver started at http://127.2.90.190:33965/ using document root <none> and password file <none>
I20260620 22:52:45.001235 12078 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:45.001278 12078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:45.001408 12078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:45.002379 12078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "016cebcbc67f4643a8d72a6a895189a6"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "ed6211a69cc33fdb4692cec830e1f120"
server_key_iv: "46e85a2b4d6ce9e7c4c53bdf7f1207d8"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.002682 12078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "016cebcbc67f4643a8d72a6a895189a6"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "ed6211a69cc33fdb4692cec830e1f120"
server_key_iv: "46e85a2b4d6ce9e7c4c53bdf7f1207d8"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.003788 12078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:45.004397 12092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.004546 12078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:45.004623 12078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "016cebcbc67f4643a8d72a6a895189a6"
format_stamp: "Formatted at 2026-06-20 22:52:44 on dist-test-slave-4mtw"
server_key: "ed6211a69cc33fdb4692cec830e1f120"
server_key_iv: "46e85a2b4d6ce9e7c4c53bdf7f1207d8"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.004704 12078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:45.007746 12078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:45.007969 12078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:45.008081 12078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:45.011494 12078 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:42555
I20260620 22:52:45.011523 12144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:42555 every 8 connection(s)
I20260620 22:52:45.011792 12078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:45.012351 12145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:45.014864 12145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6: Bootstrap starting.
I20260620 22:52:45.015092 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12078
I20260620 22:52:45.015172 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:45.015337 2410 external_mini_cluster.cc:1428] Setting key c7483b8cb6e915f16cb8e4e21acbdb0a
I20260620 22:52:45.015419 12145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:45.015756 12145 log.cc:824] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:45.016345 12145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6: No bootstrap required, opened a new log
I20260620 22:52:45.017566 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.015757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.2.90.190:42555)
0620 22:52:45.015952 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:45.015961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:45.015990 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:52:45.016381 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:45.016389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:45.016437 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:45.016566 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:45.017195 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:45.017199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:45.017199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:45.017203 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:45.017243 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:45.017253 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:45.017324 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:45.017374 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 22:52:45.017426 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260620 22:52:45.017948 12145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016cebcbc67f4643a8d72a6a895189a6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42555 } }
I20260620 22:52:45.018081 12145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:45.018123 12145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016cebcbc67f4643a8d72a6a895189a6, State: Initialized, Role: FOLLOWER
I20260620 22:52:45.018234 12145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016cebcbc67f4643a8d72a6a895189a6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42555 } }
I20260620 22:52:45.018311 12145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:45.018337 12145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:45.018366 12145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:45.019140 12145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016cebcbc67f4643a8d72a6a895189a6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42555 } }
I20260620 22:52:45.019220 12145 leader_election.cc:302] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 016cebcbc67f4643a8d72a6a895189a6; no voters:
I20260620 22:52:45.019340 12145 leader_election.cc:288] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:45.019369 12150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:45.019495 12150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [term 1 LEADER]: Becoming Leader. State: Replica: 016cebcbc67f4643a8d72a6a895189a6, State: Running, Role: LEADER
I20260620 22:52:45.019569 12145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:45.019598 12150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016cebcbc67f4643a8d72a6a895189a6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42555 } }
I20260620 22:52:45.019887 12150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 016cebcbc67f4643a8d72a6a895189a6. Latest consensus state: current_term: 1 leader_uuid: "016cebcbc67f4643a8d72a6a895189a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016cebcbc67f4643a8d72a6a895189a6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42555 } } }
I20260620 22:52:45.019968 12150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:45.019902 12152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "016cebcbc67f4643a8d72a6a895189a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016cebcbc67f4643a8d72a6a895189a6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42555 } } }
I20260620 22:52:45.020182 12152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:45.020426 12156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:45.020876 12156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:52:45.022161 12156 catalog_manager.cc:1347] Generated new cluster ID: c9f86f93ee9b4fc99f8d3a9a20768261
I20260620 22:52:45.022209 12156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:45.027650 12156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:45.027791 12156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:45.046813 12156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 016cebcbc67f4643a8d72a6a895189a6: Generated new TSK 0
I20260620 22:52:45.047010 12156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:52:45.051334 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42555
--builtin_ntp_servers=127.2.90.148:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 22:52:45.128913 12169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:45.129109 12169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:45.129137 12169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:45.130252 12169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:45.130291 12169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:45.130324 12169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:45.131642 12169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:45.131863 12169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:45.132117 12169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:45.137751 12175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:45.137800 12174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:45.137928 12177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:45.138029 12169 server_base.cc:1034] running on GCE node
I20260620 22:52:45.138212 12169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:45.138438 12169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:45.139578 12169 hybrid_clock.cc:648] HybridClock initialized: now 1781995965139566 us; error 30 us; skew 500 ppm
I20260620 22:52:45.140821 12169 webserver.cc:460] Webserver started at http://127.2.90.129:34879/ using document root <none> and password file <none>
I20260620 22:52:45.141038 12169 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:45.141084 12169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:45.141211 12169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:45.142241 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "101d7ba22ee14113a5f0ca10bb1dea22"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "9519c88f1cd241aecc38596a6ccc689a"
server_key_iv: "ca8f04929f20cc409e514a0ff6a89b58"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.142580 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "101d7ba22ee14113a5f0ca10bb1dea22"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "9519c88f1cd241aecc38596a6ccc689a"
server_key_iv: "ca8f04929f20cc409e514a0ff6a89b58"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.143784 12169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:45.144451 12183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.144608 12169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:45.144728 12169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "101d7ba22ee14113a5f0ca10bb1dea22"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "9519c88f1cd241aecc38596a6ccc689a"
server_key_iv: "ca8f04929f20cc409e514a0ff6a89b58"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.144801 12169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:45.147847 12169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:45.148053 12169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:45.148139 12169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:45.148286 12169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:45.148527 12169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:45.148554 12169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.148579 12169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:45.148597 12169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.153692 12169 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:34785
I20260620 22:52:45.153744 12296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:34785 every 8 connection(s)
I20260620 22:52:45.154006 12169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:52:45.155467 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12169
I20260620 22:52:45.155556 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:45.156040 2410 external_mini_cluster.cc:1428] Setting key bf33e2a536f86b84e612734046e642b0
I20260620 22:52:45.156788 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.154596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:41323 (local address 127.2.90.190:42555)
0620 22:52:45.154667 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:45.154669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:45.154842 (+ 173us) server_negotiation.cc:407] Connection header received
0620 22:52:45.155264 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:45.155265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:45.155270 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:45.155307 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:45.156547 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:45.156548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:45.156548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:45.156551 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:45.156563 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:45.156567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:45.156617 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:45.156710 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:52:45.156724 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:52:45.156986 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.154580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42555 (local address 127.2.90.129:41323)
0620 22:52:45.154738 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:45.154768 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:45.155179 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:45.155382 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:45.155386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:45.155396 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:45.156399 (+ 1003us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:45.156399 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:45.156443 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:45.156479 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:45.156617 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:45.156620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:45.156623 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:45.156725 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 22:52:45.156788 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260620 22:52:45.157001 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42555
--builtin_ntp_servers=127.2.90.148:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 22:52:45.157411 12297 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42555
I20260620 22:52:45.157495 12297 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:45.157627 12297 heartbeater.cc:511] Master 127.2.90.190:42555 requested a full tablet report, sending...
I20260620 22:52:45.158105 12109 ts_manager.cc:194] Registered new tserver with Master: 101d7ba22ee14113a5f0ca10bb1dea22 (127.2.90.129:34785)
W20260620 22:52:45.235105 12300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:45.235286 12300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:45.235306 12300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:45.236387 12300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:45.236419 12300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:45.236442 12300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:45.237756 12300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:45.237947 12300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:45.238179 12300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:45.242624 12305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:45.242730 12306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:45.242789 12300 server_base.cc:1034] running on GCE node
W20260620 22:52:45.242835 12308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:45.243059 12300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:45.243269 12300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:45.244410 12300 hybrid_clock.cc:648] HybridClock initialized: now 1781995965244394 us; error 33 us; skew 500 ppm
I20260620 22:52:45.245688 12300 webserver.cc:460] Webserver started at http://127.2.90.130:36279/ using document root <none> and password file <none>
I20260620 22:52:45.245929 12300 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:45.245988 12300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:45.246094 12300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:45.247115 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "ee472e6c801e4a45b9b3e6b615830cc5"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "6bb66235edef6be9779ccd8b331df178"
server_key_iv: "7e80e75672d116b94d2cc438d62d8cc4"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.247394 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "ee472e6c801e4a45b9b3e6b615830cc5"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "6bb66235edef6be9779ccd8b331df178"
server_key_iv: "7e80e75672d116b94d2cc438d62d8cc4"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.248559 12300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:52:45.249210 12314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.249352 12300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.249431 12300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "ee472e6c801e4a45b9b3e6b615830cc5"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "6bb66235edef6be9779ccd8b331df178"
server_key_iv: "7e80e75672d116b94d2cc438d62d8cc4"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.249492 12300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:45.252549 12300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:45.252799 12300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:45.252909 12300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:45.253086 12300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:45.253350 12300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:45.253379 12300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.253407 12300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:45.253437 12300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.258517 12300 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:43661
I20260620 22:52:45.258565 12427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:43661 every 8 connection(s)
I20260620 22:52:45.258836 12300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:45.260972 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12300
I20260620 22:52:45.261042 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:45.261204 2410 external_mini_cluster.cc:1428] Setting key 419c481fc7c541c35db6e7a11937db52
I20260620 22:52:45.261435 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.259428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:45869 (local address 127.2.90.190:42555)
0620 22:52:45.259471 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:45.259474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:45.259669 (+ 195us) server_negotiation.cc:407] Connection header received
0620 22:52:45.260112 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:45.260114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:45.260120 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:45.260176 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:45.261210 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:45.261212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:45.261212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:45.261217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:45.261231 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:45.261236 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:45.261284 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:45.261352 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 22:52:45.261366 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:52:45.261616 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.259417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42555 (local address 127.2.90.130:45869)
0620 22:52:45.259560 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:45.259590 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:45.260025 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:45.260241 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:45.260245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:45.260252 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:45.261067 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:45.261068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:45.261104 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:45.261133 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:45.261281 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:45.261287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:45.261290 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:45.261358 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:52:45.261414 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260620 22:52:45.261988 12428 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42555
I20260620 22:52:45.262066 12428 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:45.262100 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42555
--builtin_ntp_servers=127.2.90.148:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 22:52:45.262204 12428 heartbeater.cc:511] Master 127.2.90.190:42555 requested a full tablet report, sending...
I20260620 22:52:45.262532 12109 ts_manager.cc:194] Registered new tserver with Master: ee472e6c801e4a45b9b3e6b615830cc5 (127.2.90.130:43661)
W20260620 22:52:45.339555 12431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:45.339775 12431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:45.339807 12431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:45.341019 12431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:45.341064 12431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:45.341101 12431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:45.342478 12431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:45.342717 12431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:45.342991 12431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:45.347594 12436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:45.347718 12431 server_base.cc:1034] running on GCE node
W20260620 22:52:45.347630 12437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:45.347631 12439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:45.348023 12431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:45.348248 12431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:45.349382 12431 hybrid_clock.cc:648] HybridClock initialized: now 1781995965349360 us; error 35 us; skew 500 ppm
I20260620 22:52:45.350471 12431 webserver.cc:460] Webserver started at http://127.2.90.131:44169/ using document root <none> and password file <none>
I20260620 22:52:45.350673 12431 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:45.350718 12431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:45.350838 12431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:45.351890 12431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "62e346b45fc648e5a73adf59ce697fd9"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "637032679c45fc643be0450c746b2533"
server_key_iv: "4cbc04a92907738780b3b8505cfc6ad8"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.352207 12431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "62e346b45fc648e5a73adf59ce697fd9"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "637032679c45fc643be0450c746b2533"
server_key_iv: "4cbc04a92907738780b3b8505cfc6ad8"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.353415 12431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:52:45.354024 12445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.354164 12431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:45.354240 12431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "62e346b45fc648e5a73adf59ce697fd9"
format_stamp: "Formatted at 2026-06-20 22:52:45 on dist-test-slave-4mtw"
server_key: "637032679c45fc643be0450c746b2533"
server_key_iv: "4cbc04a92907738780b3b8505cfc6ad8"
server_key_version: "encryptionkey@0"
I20260620 22:52:45.354297 12431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:45.357513 12431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:45.357728 12431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:45.357836 12431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:45.358027 12431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:45.358287 12431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:45.358318 12431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.358345 12431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:45.358368 12431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:45.363513 12431 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:37033
I20260620 22:52:45.363548 12558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:37033 every 8 connection(s)
I20260620 22:52:45.363827 12431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:45.365976 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12431
I20260620 22:52:45.366051 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:45.366214 2410 external_mini_cluster.cc:1428] Setting key 495a184db66fd64e11ca6f265e410f19
I20260620 22:52:45.366531 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.364456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:55441 (local address 127.2.90.190:42555)
0620 22:52:45.364535 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:45.364537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:45.364724 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:52:45.365176 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:45.365177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:45.365182 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:45.365217 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:45.366293 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:45.366295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:45.366295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:52:45.366300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:45.366315 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:52:45.366319 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:45.366371 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:45.366459 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 22:52:45.366473 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:52:45.366722 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:45.364446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42555 (local address 127.2.90.131:55441)
0620 22:52:45.364606 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:45.364636 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:45.365088 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:45.365277 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:45.365280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:45.365289 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:45.366156 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:52:45.366157 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:45.366185 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:52:45.366213 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:45.366373 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:45.366377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:45.366380 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:52:45.366459 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 22:52:45.366520 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260620 22:52:45.367194 12559 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42555
I20260620 22:52:45.367278 12559 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:45.367411 12559 heartbeater.cc:511] Master 127.2.90.190:42555 requested a full tablet report, sending...
I20260620 22:52:45.367731 12109 ts_manager.cc:194] Registered new tserver with Master: 62e346b45fc648e5a73adf59ce697fd9 (127.2.90.131:37033)
I20260620 22:52:45.377818 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 22:52:45.379492 12148 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:52:45.379084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.2.90.190:42555)
0620 22:52:45.379151 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:45.379153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:45.379228 (+ 75us) server_negotiation.cc:407] Connection header received
0620 22:52:45.379278 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:45.379279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:45.379283 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:45.379318 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:45.379428 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60750: BlockingRecv error: recv got EOF from 127.0.0.1:60750 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:52:45.379556 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12169
I20260620 22:52:45.383615 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12300
I20260620 22:52:45.387928 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12431
I20260620 22:52:45.391728 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12078
2026-06-20T22:52:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (503 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:45 dist-test-slave-4mtw krb5kdc[12569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:45 dist-test-slave-4mtw krb5kdc[12569](info): set up 2 sockets
Jun 20 22:52:45 dist-test-slave-4mtw krb5kdc[12569](info): commencing operation
krb5kdc: starting...
W20260620 22:52:46.333104 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:52:46.441392 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41841
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:41841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:46.517342 12582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:46.517551 12582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:46.517580 12582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:46.518707 12582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:52:46.518747 12582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:46.518766 12582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:46.518783 12582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:52:46.518798 12582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:52:46.520224 12582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:41841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:46.520462 12582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:46.520651 12582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:46.525094 12590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:46.525192 12588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:46.525205 12582 server_base.cc:1034] running on GCE node
W20260620 22:52:46.525153 12587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:46.525542 12582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:46.525815 12582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:46.526953 12582 hybrid_clock.cc:648] HybridClock initialized: now 1781995966526936 us; error 31 us; skew 500 ppm
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:46.532945 12582 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:52:46.533360 12582 webserver.cc:460] Webserver started at http://127.2.90.190:34331/ using document root <none> and password file <none>
I20260620 22:52:46.533550 12582 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:46.533594 12582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:46.533713 12582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:46.534704 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "3a171670b82746c5ad525d4cda18b6b6"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "18adb377a6a7e86720caea03be9234e5"
server_key_iv: "f1812d4e6f38869cac4d362e56d81811"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.535025 12582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "3a171670b82746c5ad525d4cda18b6b6"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "18adb377a6a7e86720caea03be9234e5"
server_key_iv: "f1812d4e6f38869cac4d362e56d81811"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.536199 12582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:46.536835 12597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.536990 12582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.537070 12582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "3a171670b82746c5ad525d4cda18b6b6"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "18adb377a6a7e86720caea03be9234e5"
server_key_iv: "f1812d4e6f38869cac4d362e56d81811"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.537135 12582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:46.555217 12582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:46.557277 12582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:46.557410 12582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:46.561079 12582 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:41841
I20260620 22:52:46.561120 12649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:41841 every 8 connection(s)
I20260620 22:52:46.561430 12582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:52:46.562314 12650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:46.564442 12650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6: Bootstrap starting.
I20260620 22:52:46.564975 12650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:46.565210 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12582
I20260620 22:52:46.565263 12650 log.cc:824] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:46.565291 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:52:46.565495 2410 external_mini_cluster.cc:1428] Setting key 3287995d8c8dc24d0ae0c02994b81ecf
I20260620 22:52:46.566022 12650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6: No bootstrap required, opened a new log
I20260620 22:52:46.568554 12650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a171670b82746c5ad525d4cda18b6b6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41841 } }
I20260620 22:52:46.568742 12650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:46.568781 12650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a171670b82746c5ad525d4cda18b6b6, State: Initialized, Role: FOLLOWER
I20260620 22:52:46.568908 12650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a171670b82746c5ad525d4cda18b6b6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41841 } }
I20260620 22:52:46.568990 12650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:52:46.569021 12650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:52:46.569067 12650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:46.569612 12650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a171670b82746c5ad525d4cda18b6b6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41841 } }
I20260620 22:52:46.569713 12650 leader_election.cc:302] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a171670b82746c5ad525d4cda18b6b6; no voters:
I20260620 22:52:46.569852 12650 leader_election.cc:288] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:52:46.569895 12655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:46.570055 12655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3a171670b82746c5ad525d4cda18b6b6, State: Running, Role: LEADER
I20260620 22:52:46.570076 12650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:52:46.570145 12655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a171670b82746c5ad525d4cda18b6b6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41841 } }
I20260620 22:52:46.570485 12655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a171670b82746c5ad525d4cda18b6b6. Latest consensus state: current_term: 1 leader_uuid: "3a171670b82746c5ad525d4cda18b6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a171670b82746c5ad525d4cda18b6b6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41841 } } }
I20260620 22:52:46.570586 12655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:46.570735 12656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a171670b82746c5ad525d4cda18b6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a171670b82746c5ad525d4cda18b6b6" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41841 } } }
I20260620 22:52:46.570801 12656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6 [sys.catalog]: This master's current role is: LEADER
I20260620 22:52:46.570926 12664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:52:46.571405 12664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:46.572908 12664 catalog_manager.cc:1347] Generated new cluster ID: a3248dd0ec494d9eab22ed8af940307f
I20260620 22:52:46.572961 12664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:52:46.575723 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:46.565943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.2.90.190:41841)
0620 22:52:46.566101 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:46.566111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:46.566149 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:52:46.566607 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:46.566616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:46.566684 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:46.566833 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:46.567344 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.568673 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.569470 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.570061 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.570479 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:46.572835 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:46.572839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:46.572839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:46.572850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:46.574377 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:46.574698 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:46.574701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:46.574701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:46.574780 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:46.574952 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:46.574953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:46.574953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:46.575171 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:46.575319 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:46.575403 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 22:52:46.575530 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260620 22:52:46.580624 12664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:52:46.581346 12664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:52:46.585335 12664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6: Generated new TSK 0
I20260620 22:52:46.585476 12664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:52:46.611267 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41841
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:46.701076 12676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:46.701264 12676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:46.701285 12676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:46.702371 12676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:46.702410 12676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:46.702442 12676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:52:46.703800 12676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:46.703995 12676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:46.704196 12676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:46.708726 12682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:46.708719 12681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:46.708866 12676 server_base.cc:1034] running on GCE node
W20260620 22:52:46.708974 12684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:46.709168 12676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:46.709414 12676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:46.710551 12676 hybrid_clock.cc:648] HybridClock initialized: now 1781995966710539 us; error 31 us; skew 500 ppm
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:46.712966 12676 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:52:46.713368 12676 webserver.cc:460] Webserver started at http://127.2.90.129:39971/ using document root <none> and password file <none>
I20260620 22:52:46.713554 12676 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:46.713600 12676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:46.713719 12676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:46.714720 12676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "47b4b84a0cf04c2fb18f364038448e78"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "c3645917a36381e09637475564fa7ec4"
server_key_iv: "8ec55b3177ca1e6f0e15fcd57c01c4d6"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.715045 12676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "47b4b84a0cf04c2fb18f364038448e78"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "c3645917a36381e09637475564fa7ec4"
server_key_iv: "8ec55b3177ca1e6f0e15fcd57c01c4d6"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.716208 12676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:52:46.716876 12691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.717031 12676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.717118 12676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "47b4b84a0cf04c2fb18f364038448e78"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "c3645917a36381e09637475564fa7ec4"
server_key_iv: "8ec55b3177ca1e6f0e15fcd57c01c4d6"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.717177 12676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:46.738214 12676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:46.740377 12676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:46.740513 12676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:46.740828 12676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:46.741125 12676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:46.741158 12676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.741180 12676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:46.741204 12676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.746629 12676 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:36609
I20260620 22:52:46.746696 12804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:36609 every 8 connection(s)
I20260620 22:52:46.746974 12676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:46.754823 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:46.747611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:52349 (local address 127.2.90.190:41841)
0620 22:52:46.747653 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:46.747655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:46.747844 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:52:46.748303 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:46.748304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:46.748314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:46.748364 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:46.749228 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.749723 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.750616 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.750804 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.750847 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:46.752781 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:46.752783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:46.752783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:46.752790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:46.753831 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:46.754153 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:46.754154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:46.754154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:46.754197 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:46.754361 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:46.754362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:46.754362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:46.754462 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:46.754505 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:46.754703 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 22:52:46.754761 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:52:46.755016 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:46.747591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41841 (local address 127.2.90.129:52349)
0620 22:52:46.747723 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:46.747758 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:46.748228 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:46.748420 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:46.748424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:46.748977 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:46.749165 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:46.749170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.749786 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.749787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:46.750556 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:46.750558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.750856 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.750856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:46.750957 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:46.751406 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:46.751415 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:46.752696 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:46.753893 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:46.753897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:46.753899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:46.754101 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:46.754240 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:46.754241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:46.754241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:46.754322 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:46.754504 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:46.754507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:46.754613 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 22:52:46.754688 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 22:52:46.754798 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260620 22:52:46.755472 12805 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41841
I20260620 22:52:46.755558 12805 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:46.755725 12805 heartbeater.cc:511] Master 127.2.90.190:41841 requested a full tablet report, sending...
I20260620 22:52:46.755730 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12676
I20260620 22:52:46.755812 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:52:46.756019 2410 external_mini_cluster.cc:1428] Setting key e94e733d8949abcabc1d6d7f4ed054ee
I20260620 22:52:46.756201 12614 ts_manager.cc:194] Registered new tserver with Master: 47b4b84a0cf04c2fb18f364038448e78 (127.2.90.129:36609)
I20260620 22:52:46.757162 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:52349
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:52:46.776947 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41841
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:46.853434 12810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:46.853615 12810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:46.853636 12810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:46.854683 12810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:46.854715 12810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:46.854738 12810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:52:46.856067 12810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:46.856231 12810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:46.856401 12810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:46.860775 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:46.860832 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:46.860776 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:46.860946 12810 server_base.cc:1034] running on GCE node
I20260620 22:52:46.861133 12810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:46.861338 12810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:46.862468 12810 hybrid_clock.cc:648] HybridClock initialized: now 1781995966862456 us; error 28 us; skew 500 ppm
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:46.864709 12810 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:52:46.865123 12810 webserver.cc:460] Webserver started at http://127.2.90.130:41379/ using document root <none> and password file <none>
I20260620 22:52:46.865311 12810 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:46.865356 12810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:46.865486 12810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:46.866426 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "20700ff7202d49649a04099e815a0dc2"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "5099e358a77d0764fddf4b0ed549a55f"
server_key_iv: "f751439c8a68c6258a66819dbe1555a5"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.866741 12810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "20700ff7202d49649a04099e815a0dc2"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "5099e358a77d0764fddf4b0ed549a55f"
server_key_iv: "f751439c8a68c6258a66819dbe1555a5"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.867914 12810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:52:46.868541 12825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.868700 12810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:52:46.868788 12810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "20700ff7202d49649a04099e815a0dc2"
format_stamp: "Formatted at 2026-06-20 22:52:46 on dist-test-slave-4mtw"
server_key: "5099e358a77d0764fddf4b0ed549a55f"
server_key_iv: "f751439c8a68c6258a66819dbe1555a5"
server_key_version: "encryptionkey@0"
I20260620 22:52:46.868846 12810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:46.887801 12810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:46.889910 12810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:46.890050 12810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:46.890252 12810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:46.890548 12810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:46.890583 12810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.890610 12810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:46.890627 12810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:46.895888 12810 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:41403
I20260620 22:52:46.895942 12938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:41403 every 8 connection(s)
I20260620 22:52:46.896224 12810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:52:46.900955 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12810
I20260620 22:52:46.901052 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:52:46.901248 2410 external_mini_cluster.cc:1428] Setting key 7ab3c9728d572d4ed7f56124ff638f75
Jun 20 22:52:46 dist-test-slave-4mtw krb5kdc[12569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995966, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:46.905162 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:46.896867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:44893 (local address 127.2.90.190:41841)
0620 22:52:46.896922 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:46.896924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:46.897098 (+ 174us) server_negotiation.cc:407] Connection header received
0620 22:52:46.897598 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:46.897599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:46.897611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:46.897661 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:46.898551 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.899041 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.899993 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.900185 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.900238 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:46.902396 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:46.902398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:46.902398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:46.902405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:46.903733 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:46.904294 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:46.904295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:46.904295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:46.904356 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:46.904569 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:46.904570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:46.904570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:46.904720 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:46.904781 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:46.904998 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 22:52:46.905075 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:52:46.905257 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:46.896847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41841 (local address 127.2.90.130:44893)
0620 22:52:46.896983 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:46.897022 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:46.897502 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:46.897734 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:46.897738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:46.898274 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:46.898460 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:46.898464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.899118 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.899119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:46.899907 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:46.899909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:46.900248 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:46.900249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:46.900369 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:46.900809 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:46.900819 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:46.902316 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:46.903802 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:46.903835 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:46.903838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:46.904212 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:46.904423 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:46.904423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:46.904424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:46.904520 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:46.904768 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:46.904771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:46.904881 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 22:52:46.904962 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:52:46.905077 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260620 22:52:46.905803 12939 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41841
I20260620 22:52:46.905901 12939 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:46.906092 12939 heartbeater.cc:511] Master 127.2.90.190:41841 requested a full tablet report, sending...
I20260620 22:52:46.906453 12614 ts_manager.cc:194] Registered new tserver with Master: 20700ff7202d49649a04099e815a0dc2 (127.2.90.130:41403)
I20260620 22:52:46.906940 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:44893
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:52:46.923022 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41841
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:52:46.999367 12944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:52:46.999569 12944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:52:46.999590 12944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:52:47.000710 12944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:52:47.000748 12944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:52:47.000774 12944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:52:47.002173 12944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:52:47.002389 12944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:52:47.002599 12944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:52:47.007221 12949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:47.007244 12950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:52:47.007259 12952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:52:47.007791 12944 server_base.cc:1034] running on GCE node
I20260620 22:52:47.007982 12944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:52:47.008227 12944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:52:47.009373 12944 hybrid_clock.cc:648] HybridClock initialized: now 1781995967009358 us; error 29 us; skew 500 ppm
Jun 20 22:52:47 dist-test-slave-4mtw krb5kdc[12569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995967, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:52:47.011744 12944 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:52:47.012141 12944 webserver.cc:460] Webserver started at http://127.2.90.131:36555/ using document root <none> and password file <none>
I20260620 22:52:47.012344 12944 fs_manager.cc:359] Metadata directory not provided
I20260620 22:52:47.012382 12944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:52:47.012476 12944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:52:47.013476 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "5e6b2c7545ab469895eae187ae85916d"
format_stamp: "Formatted at 2026-06-20 22:52:47 on dist-test-slave-4mtw"
server_key: "6a79d69f243118870ffb6ce5fb604c42"
server_key_iv: "3090a3cfcbea02740b066946f7871d21"
server_key_version: "encryptionkey@0"
I20260620 22:52:47.013754 12944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "5e6b2c7545ab469895eae187ae85916d"
format_stamp: "Formatted at 2026-06-20 22:52:47 on dist-test-slave-4mtw"
server_key: "6a79d69f243118870ffb6ce5fb604c42"
server_key_iv: "3090a3cfcbea02740b066946f7871d21"
server_key_version: "encryptionkey@0"
I20260620 22:52:47.014855 12944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:52:47.015453 12959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:47.015591 12944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:52:47.015661 12944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "5e6b2c7545ab469895eae187ae85916d"
format_stamp: "Formatted at 2026-06-20 22:52:47 on dist-test-slave-4mtw"
server_key: "6a79d69f243118870ffb6ce5fb604c42"
server_key_iv: "3090a3cfcbea02740b066946f7871d21"
server_key_version: "encryptionkey@0"
I20260620 22:52:47.015719 12944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:52:47.025231 12944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:52:47.026823 12944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:52:47.026952 12944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:52:47.027139 12944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:52:47.027407 12944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:52:47.027439 12944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:47.027467 12944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:52:47.027489 12944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:52:47.032716 12944 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:43091
I20260620 22:52:47.032784 13072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:43091 every 8 connection(s)
I20260620 22:52:47.033043 12944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:52:47.036880 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 12944
I20260620 22:52:47.036976 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:52:47.037149 2410 external_mini_cluster.cc:1428] Setting key 4053fcb50e1b32ad25d146cfd14a6668
Jun 20 22:52:47 dist-test-slave-4mtw krb5kdc[12569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995967, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:52:47.041549 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.033659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:55329 (local address 127.2.90.190:41841)
0620 22:52:47.033702 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.033704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.033878 (+ 174us) server_negotiation.cc:407] Connection header received
0620 22:52:47.034372 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.034373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.034385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.034435 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:47.035307 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.035794 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.036798 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.037073 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.037107 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.039061 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:47.039062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:47.039063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:47.039070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:47.040343 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:47.040722 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:47.040723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:47.040723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:47.040779 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:47.040985 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:47.040986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:47.040986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:47.041095 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:47.041149 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.041386 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.041464 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:52:47.041703 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.033623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41841 (local address 127.2.90.131:55329)
0620 22:52:47.033768 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.033803 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:47.034275 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:47.034506 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:47.034510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:47.035031 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:52:47.035215 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.035219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.035870 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.035872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.036666 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.036668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.037176 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.037176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.037326 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.037778 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:52:47.037786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:52:47.038982 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:52:47.040424 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:47.040431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:47.040433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:47.040637 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:47.040830 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:52:47.040831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:52:47.040831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:52:47.040934 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:52:47.041146 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:52:47.041151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:52:47.041276 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 22:52:47.041359 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:52:47.041509 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260620 22:52:47.042068 13073 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41841
I20260620 22:52:47.042152 13073 heartbeater.cc:464] Registering TS with master...
I20260620 22:52:47.042305 13073 heartbeater.cc:511] Master 127.2.90.190:41841 requested a full tablet report, sending...
I20260620 22:52:47.042599 12614 ts_manager.cc:194] Registered new tserver with Master: 5e6b2c7545ab469895eae187ae85916d (127.2.90.131:43091)
I20260620 22:52:47.042922 12614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:55329
I20260620 22:52:47.048544 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:52:47.054602 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.049747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.2.90.190:41841)
0620 22:52:47.049783 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.049785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.049873 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:52:47.049910 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.049911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.049928 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.049969 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:52:47.050420 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.050867 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.051506 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.051669 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.051707 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.052646 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:52:47.052647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:52:47.052647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:52:47.052653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:52:47.053803 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:47.054048 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:47.054049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:47.054049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:47.054092 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:52:47.054198 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:52:47.054198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:52:47.054199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:52:47.054317 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:52:47.054382 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.054466 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.054525 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:52:47.055892 12614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:52:47.057479 12614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:52:47.063176 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.059364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:43091 (local address 127.0.0.1:43536)
0620 22:52:47.059470 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.059511 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:47.059570 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:47.060107 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:47.060108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:47.060110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:47.060224 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.060225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.061480 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.061481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.062224 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.062225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.062964 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.062965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.063022 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.063033 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 22:52:47.063084 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:52:47.063120 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":79,"thread_start_us":79,"threads_started":1}
I20260620 22:52:47.063212 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.059536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.2.90.131:43091)
0620 22:52:47.059903 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.059907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.059930 (+ 23us) server_negotiation.cc:407] Connection header received
0620 22:52:47.059964 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.059966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.059981 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.060087 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:47.060347 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.061397 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.062297 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.062881 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.062964 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.063019 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.063095 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.063133 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":77,"threads_started":1}
I20260620 22:52:47.063962 13007 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8618bf55854149bb9baf885abe3ccb (DEFAULT_TABLE table=test-table [id=6c8b570d77af4e5dbc1d03e45f7a84c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:47.064337 13007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8618bf55854149bb9baf885abe3ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:47.064342 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.059568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:41403 (local address 127.0.0.1:33766)
0620 22:52:47.059686 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.059696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:47.059731 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:47.060111 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:47.060111 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:47.060113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:47.060167 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.060168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.061200 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.061201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.062496 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.062497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.064127 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.064128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.064188 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.064192 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:47.064244 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:52:47.064279 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260620 22:52:47.064369 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.059800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.2.90.130:41403)
0620 22:52:47.059963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.059967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.059975 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:47.060008 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.060010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.060024 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.060306 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:47.060372 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.061117 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.063399 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.064053 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.064127 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.064182 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.064260 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.064303 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260620 22:52:47.064503 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.059369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36609 (local address 127.0.0.1:44782)
0620 22:52:47.059477 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.059513 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:47.059570 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:47.060005 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:47.060008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:47.060010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:47.060079 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.060080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.061362 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.061363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.062489 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.062492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.064219 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.064220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.064281 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.064285 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:52:47.064333 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:52:47.064374 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":5,"thread_start_us":55,"threads_started":1}
I20260620 22:52:47.064736 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.059711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.2.90.129:36609)
0620 22:52:47.059859 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.059862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.059870 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:52:47.059907 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.059909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.059924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.059997 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:47.060157 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.061261 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.062575 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.063253 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.063317 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.063369 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.064596 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.064642 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260620 22:52:47.065045 12873 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8618bf55854149bb9baf885abe3ccb (DEFAULT_TABLE table=test-table [id=6c8b570d77af4e5dbc1d03e45f7a84c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:47.065323 12873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8618bf55854149bb9baf885abe3ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:47.067077 13087 tablet_bootstrap.cc:492] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Bootstrap starting.
I20260620 22:52:47.067750 13087 tablet_bootstrap.cc:654] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:47.067755 13088 tablet_bootstrap.cc:492] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Bootstrap starting.
I20260620 22:52:47.068306 13088 tablet_bootstrap.cc:654] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:47.068202 12739 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8618bf55854149bb9baf885abe3ccb (DEFAULT_TABLE table=test-table [id=6c8b570d77af4e5dbc1d03e45f7a84c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:52:47.068311 13087 log.cc:824] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:47.068512 12739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8618bf55854149bb9baf885abe3ccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:52:47.068814 13088 log.cc:824] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:47.069064 13087 tablet_bootstrap.cc:492] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: No bootstrap required, opened a new log
I20260620 22:52:47.069135 13087 ts_tablet_manager.cc:1397] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:47.069442 13088 tablet_bootstrap.cc:492] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: No bootstrap required, opened a new log
I20260620 22:52:47.069499 13088 ts_tablet_manager.cc:1397] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:52:47.071008 13088 raft_consensus.cc:348] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.071098 13087 raft_consensus.cc:348] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.071277 13087 raft_consensus.cc:374] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:47.071326 13088 raft_consensus.cc:374] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:47.071420 13088 raft_consensus.cc:729] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20700ff7202d49649a04099e815a0dc2, State: Initialized, Role: FOLLOWER
I20260620 22:52:47.071547 13091 tablet_bootstrap.cc:492] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78: Bootstrap starting.
I20260620 22:52:47.071530 13088 consensus_queue.cc:260] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.071401 13087 raft_consensus.cc:729] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6b2c7545ab469895eae187ae85916d, State: Initialized, Role: FOLLOWER
I20260620 22:52:47.071792 13088 ts_tablet_manager.cc:1428] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:52:47.071768 13087 consensus_queue.cc:260] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.071918 12939 heartbeater.cc:503] Master 127.2.90.190:41841 was elected leader, sending a full tablet report...
I20260620 22:52:47.072116 13091 tablet_bootstrap.cc:654] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78: Neither blocks nor log segments found. Creating new log.
I20260620 22:52:47.072295 13087 ts_tablet_manager.cc:1428] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:52:47.072311 13073 heartbeater.cc:503] Master 127.2.90.190:41841 was elected leader, sending a full tablet report...
I20260620 22:52:47.072659 13091 log.cc:824] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78: Log is configured to *not* fsync() on all Append() calls
I20260620 22:52:47.073400 13091 tablet_bootstrap.cc:492] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78: No bootstrap required, opened a new log
I20260620 22:52:47.073457 13091 ts_tablet_manager.cc:1397] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:52:47.074681 13091 raft_consensus.cc:348] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.074803 13091 raft_consensus.cc:374] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:52:47.074831 13091 raft_consensus.cc:729] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b4b84a0cf04c2fb18f364038448e78, State: Initialized, Role: FOLLOWER
I20260620 22:52:47.074905 13091 consensus_queue.cc:260] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.075071 13091 ts_tablet_manager.cc:1428] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:52:47.075107 12805 heartbeater.cc:503] Master 127.2.90.190:41841 was elected leader, sending a full tablet report...
W20260620 22:52:47.146735 12940 tablet.cc:2369] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:52:47.211369 13095 raft_consensus.cc:482] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:52:47.211483 13095 raft_consensus.cc:504] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.211843 13095 leader_election.cc:288] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e6b2c7545ab469895eae187ae85916d (127.2.90.131:43091), 20700ff7202d49649a04099e815a0dc2 (127.2.90.130:41403)
I20260620 22:52:47.215557 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.212070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:41863 (local address 127.2.90.131:43091)
0620 22:52:47.212134 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.212136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.212146 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:52:47.212251 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.212252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.212261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.212309 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:47.212458 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.213237 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.214655 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.215223 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.215289 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.215297 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.215453 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.215488 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:52:47.215557 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.212210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:58845 (local address 127.2.90.130:41403)
0620 22:52:47.212271 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.212272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.212385 (+ 113us) server_negotiation.cc:407] Connection header received
0620 22:52:47.212431 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.212433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.212444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.212488 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:52:47.212596 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.213280 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.214657 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.215219 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.215289 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.215297 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.215453 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.215488 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 22:52:47.215574 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.212018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:43091 (local address 127.2.90.129:41863)
0620 22:52:47.212067 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.212079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:47.212141 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:47.212321 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:47.212322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:47.212325 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:47.212396 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.212399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.213310 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.213310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.214575 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.214578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.215293 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.215293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.215388 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.215390 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:52:47.215448 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:52:47.215480 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":27}
I20260620 22:52:47.215579 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.212122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:41403 (local address 127.2.90.129:58845)
0620 22:52:47.212305 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.212314 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:52:47.212373 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:52:47.212502 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:52:47.212503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:52:47.212504 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:52:47.212548 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.212549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.213340 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.213340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.214575 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:52:47.214578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.215293 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.215293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:52:47.215384 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.215389 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:52:47.215441 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:52:47.215480 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":34,"thread_start_us":127,"threads_started":1}
I20260620 22:52:47.215904 13027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8618bf55854149bb9baf885abe3ccb" candidate_uuid: "47b4b84a0cf04c2fb18f364038448e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6b2c7545ab469895eae187ae85916d" is_pre_election: true
I20260620 22:52:47.216068 13027 raft_consensus.cc:2454] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b4b84a0cf04c2fb18f364038448e78 in term 0.
I20260620 22:52:47.216035 12893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8618bf55854149bb9baf885abe3ccb" candidate_uuid: "47b4b84a0cf04c2fb18f364038448e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20700ff7202d49649a04099e815a0dc2" is_pre_election: true
I20260620 22:52:47.216187 12893 raft_consensus.cc:2454] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b4b84a0cf04c2fb18f364038448e78 in term 0.
I20260620 22:52:47.216266 12692 leader_election.cc:302] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b4b84a0cf04c2fb18f364038448e78, 5e6b2c7545ab469895eae187ae85916d; no voters:
I20260620 22:52:47.216389 13095 raft_consensus.cc:2781] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:52:47.216456 13095 raft_consensus.cc:482] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:52:47.216475 13095 raft_consensus.cc:3037] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:47.217186 13095 raft_consensus.cc:504] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.217294 13095 leader_election.cc:288] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [CANDIDATE]: Term 1 election: Requested vote from peers 5e6b2c7545ab469895eae187ae85916d (127.2.90.131:43091), 20700ff7202d49649a04099e815a0dc2 (127.2.90.130:41403)
I20260620 22:52:47.217463 13027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8618bf55854149bb9baf885abe3ccb" candidate_uuid: "47b4b84a0cf04c2fb18f364038448e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6b2c7545ab469895eae187ae85916d"
I20260620 22:52:47.217478 12893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8618bf55854149bb9baf885abe3ccb" candidate_uuid: "47b4b84a0cf04c2fb18f364038448e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20700ff7202d49649a04099e815a0dc2"
I20260620 22:52:47.217536 12893 raft_consensus.cc:3037] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:47.217540 13027 raft_consensus.cc:3037] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:52:47.218242 12893 raft_consensus.cc:2454] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b4b84a0cf04c2fb18f364038448e78 in term 1.
I20260620 22:52:47.218245 13027 raft_consensus.cc:2454] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b4b84a0cf04c2fb18f364038448e78 in term 1.
I20260620 22:52:47.218382 12692 leader_election.cc:302] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b4b84a0cf04c2fb18f364038448e78, 5e6b2c7545ab469895eae187ae85916d; no voters:
I20260620 22:52:47.218462 13095 raft_consensus.cc:2781] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:52:47.218559 13095 raft_consensus.cc:686] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [term 1 LEADER]: Becoming Leader. State: Replica: 47b4b84a0cf04c2fb18f364038448e78, State: Running, Role: LEADER
I20260620 22:52:47.218639 13095 consensus_queue.cc:237] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } }
I20260620 22:52:47.219391 12614 catalog_manager.cc:5495] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b4b84a0cf04c2fb18f364038448e78 (127.2.90.129). New cstate: current_term: 1 leader_uuid: "47b4b84a0cf04c2fb18f364038448e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b4b84a0cf04c2fb18f364038448e78" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 } health_report { overall_health: UNKNOWN } } }
I20260620 22:52:47.242304 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.240178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.2.90.129:36609)
0620 22:52:47.240223 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:52:47.240224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:52:47.240246 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:52:47.240278 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:52:47.240279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:52:47.240298 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:52:47.240353 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:52:47.240499 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.241011 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.241719 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:52:47.241893 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:52:47.241937 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:52:47.242040 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:52:47.242139 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:52:47.242195 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:52:47.242210 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 22:52:47.242227 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:52:47.243935 13027 raft_consensus.cc:1261] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 1 FOLLOWER]: Refusing update from remote peer 47b4b84a0cf04c2fb18f364038448e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:47.243955 12893 raft_consensus.cc:1261] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 1 FOLLOWER]: Refusing update from remote peer 47b4b84a0cf04c2fb18f364038448e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:52:47.244253 13097 consensus_queue.cc:1035] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20700ff7202d49649a04099e815a0dc2" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:47.244372 13095 consensus_queue.cc:1035] T 7f8618bf55854149bb9baf885abe3ccb P 47b4b84a0cf04c2fb18f364038448e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6b2c7545ab469895eae187ae85916d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:52:47.246789 13100 mvcc.cc:205] Tried to move back new op lower bound from 7299055481828921344 to 7299055481728036864. Current Snapshot: MvccSnapshot[applied={T|T < 7299055481828921344 or (T in {7299055481828921344})}]
I20260620 22:52:47.246897 13102 mvcc.cc:205] Tried to move back new op lower bound from 7299055481828921344 to 7299055481728036864. Current Snapshot: MvccSnapshot[applied={T|T < 7299055481828921344}]
I20260620 22:52:47.247711 13101 mvcc.cc:205] Tried to move back new op lower bound from 7299055481828921344 to 7299055481728036864. Current Snapshot: MvccSnapshot[applied={T|T < 7299055481828921344}]
I20260620 22:52:47.248593 12613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:47.248713 12613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:52:47.249418 12613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a171670b82746c5ad525d4cda18b6b6: Sending DeleteTablet for 3 replicas of tablet 7f8618bf55854149bb9baf885abe3ccb
I20260620 22:52:47.249697 13007 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8618bf55854149bb9baf885abe3ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:47 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:43536
I20260620 22:52:47.249760 12873 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8618bf55854149bb9baf885abe3ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:47 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33766
I20260620 22:52:47.249972 13110 tablet_replica.cc:330] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: stopping tablet replica
I20260620 22:52:47.249958 12739 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8618bf55854149bb9baf885abe3ccb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:52:47 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:44782
I20260620 22:52:47.250098 13110 raft_consensus.cc:2229] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:47.250154 13111 tablet_replica.cc:330] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: stopping tablet replica
I20260620 22:52:47.250201 13110 raft_consensus.cc:2258] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:47.250234 13111 raft_consensus.cc:2229] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:52:47.250237 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12676
I20260620 22:52:47.250319 13111 raft_consensus.cc:2258] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:52:47.250675 13110 ts_tablet_manager.cc:1905] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:47.251776 13111 ts_tablet_manager.cc:1905] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:52:47.252099 13110 ts_tablet_manager.cc:1918] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:47.252144 13110 log.cc:1196] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/7f8618bf55854149bb9baf885abe3ccb
I20260620 22:52:47.252326 13110 ts_tablet_manager.cc:1939] T 7f8618bf55854149bb9baf885abe3ccb P 5e6b2c7545ab469895eae187ae85916d: Deleting consensus metadata
I20260620 22:52:47.252624 12598 catalog_manager.cc:4841] TS 5e6b2c7545ab469895eae187ae85916d (127.2.90.131:43091): tablet 7f8618bf55854149bb9baf885abe3ccb (table test-table [id=6c8b570d77af4e5dbc1d03e45f7a84c5]) successfully deleted
I20260620 22:52:47.253189 13111 ts_tablet_manager.cc:1918] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:52:47.253252 13111 log.cc:1196] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/7f8618bf55854149bb9baf885abe3ccb
I20260620 22:52:47.253456 13111 ts_tablet_manager.cc:1939] T 7f8618bf55854149bb9baf885abe3ccb P 20700ff7202d49649a04099e815a0dc2: Deleting consensus metadata
I20260620 22:52:47.253795 12598 catalog_manager.cc:4841] TS 20700ff7202d49649a04099e815a0dc2 (127.2.90.130:41403): tablet 7f8618bf55854149bb9baf885abe3ccb (table test-table [id=6c8b570d77af4e5dbc1d03e45f7a84c5]) successfully deleted
W20260620 22:52:47.256623 12599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:36609 (error 108)
I20260620 22:52:47.256860 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12810
I20260620 22:52:47.257400 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:52:47.256959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:36609 (local address 127.0.0.1:44790)
0620 22:52:47.257337 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 22:52:47.257373 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:36609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
W20260620 22:52:47.257520 12599 catalog_manager.cc:4584] TS 47b4b84a0cf04c2fb18f364038448e78 (127.2.90.129:36609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f8618bf55854149bb9baf885abe3ccb: Network error: Client connection negotiation failed: client connection to 127.2.90.129:36609: connect: Connection refused (error 111)
I20260620 22:52:47.261439 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12944
I20260620 22:52:47.264906 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 12582
2026-06-20T22:52:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1874 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:52:47 dist-test-slave-4mtw krb5kdc[13116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:52:47 dist-test-slave-4mtw krb5kdc[13116](info): set up 2 sockets
Jun 20 22:52:47 dist-test-slave-4mtw krb5kdc[13116](info): commencing operation
krb5kdc: starting...
W20260620 22:52:48.201160 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:52:48 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:52:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/rangeradmin_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/rangeradmin_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/rangerlookup_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/rangerlookup_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
I20260620 22:52:48.344954 2410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5TG4FZ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/postgres -l logfile start
2026-06-20 22:52:50.534 UTC [13146] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:52:50.534 UTC [13146] LOG: listening on IPv4 address "127.2.90.148", port 43331
2026-06-20 22:52:50.537 UTC [13146] LOG: listening on Unix socket "/tmp/.s.PGSQL.43331"
2026-06-20 22:52:50.562 UTC [13149] LOG: database system was shut down at 2026-06-20 22:52:49 UTC
2026-06-20 22:52:50.565 UTC [13146] LOG: database system is ready to accept connections
I20260620 22:52:51.445281 2410 mini_postgres.cc:96] Postgres bound to 43331
2026-06-20 22:52:51.450 UTC [13157] FATAL: database "slave" does not exist
127.2.90.148:43331 - accepting connections
I20260620 22:52:51.451308 2410 mini_ranger.cc:161] Starting Ranger...
I20260620 22:52:51.464334 2410 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:52:51.850226 2410 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:52:51.850302 2410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-admin
2026-06-20 22:52:52,166 [I] DB FLAVOR :POSTGRES
2026-06-20 22:52:52,167 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:52:52,167 [I] Checking connection..
2026-06-20 22:52:52,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:52:52,432 [I] Checking connection passed.
2026-06-20 22:52:52,433 [I] --------- Verifying version history table ---------
2026-06-20 22:52:52,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:52:52,663 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:52:52,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:52:52,894 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:52:52,894 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:52:52,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:52:53.138 UTC [13221] WARNING: there is no transaction in progress
2026-06-20 22:52:53,143 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:52:53,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:52:53,380 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:52:53,380 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:52:53,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:52:53,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:52:53,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4mtw.c.gcp-upstream.internal','N') ;"
2026-06-20 22:52:54,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:52:54,326 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:52:54,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:52:54,578 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:52:54,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:52:54,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:52:55,061 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:52:55,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:52:55.922 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:55.939 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:55.954 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:55.968 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:55.982 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:55.997 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.047 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.053 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.066 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.076 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.085 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.098 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.106 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.112 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.118 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56.124 UTC [13329] WARNING: there is no transaction in progress
2026-06-20 22:52:56,672 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:52:56,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:52:56,895 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:52:56,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:52:57,159 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:52:57,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:52:57,399 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:52:57,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:52:57,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:52:57,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4mtw.c.gcp-upstream.internal';"
2026-06-20 22:52:58,120 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:52:58,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:52:58,352 [I] DB_PATCHES have already been applied
I20260620 22:52:58.357331 2410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-admin:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5TG4FZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:52:58.357391 2410 mini_ranger.cc:193] Using host: 127.2.90.148
I20260620 22:52:58.360090 2410 mini_ranger.cc:227] Ranger admin URL: http://127.2.90.148:33697
Jun 20, 2026 10:52:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:52:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:52:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:52:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:52:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:52:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:52:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:52:59.388212 2410 mini_ranger.cc:160] Time spent starting Ranger: real 7.937s user 0.000s sys 0.007s
Jun 20, 2026 10:53:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:53:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 10:53:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:53:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:53:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:53:19.925354 2410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/rangerkms_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/rangerkms_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.90.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.90.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:53:19 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995999, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:53:20.006860 2410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:53:20.015916 2410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:53:20.698040 2410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:53:20.698120 2410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms
I20260620 22:53:20.698136 2410 mini_ranger_kms.cc:227] Using postgres at 127.2.90.148:43331
2026-06-20 22:53:20,866 [I] DB FLAVOR :POSTGRES
2026-06-20 22:53:20,867 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:53:20,867 [I] Checking connection
2026-06-20 22:53:20,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:53:21,094 [I] connection success
2026-06-20 22:53:21,094 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:53:21,094 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:53:21,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:53:21,317 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:53:21,317 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:53:21,317 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:53:21,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:43331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:53:21,565 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:53:22.008340 2410 mini_ranger_kms.cc:310] Created kms service
I20260620 22:53:22.169312 2410 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:53:22.207144 2410 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:53:22 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781995999, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:22.544667 2410 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:53:22.544821 2410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms:/tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:53:22.544847 2410 mini_ranger_kms.cc:240] Using host: 127.2.90.148
I20260620 22:53:22.548243 2410 mini_ranger_kms.cc:276] Ranger KMS PID: 13509
I20260620 22:53:22.548341 2410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.90.148:41371
Jun 20, 2026 10:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 22:53:24.556284 2410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.549s user 0.000s sys 0.150s
I20260620 22:53:24.556392 2410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:53:24.556474 2410 mini_ranger_kms.cc:401] 127.2.90.148:41371/kms/v1/keys
Jun 20 22:53:24 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.90.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 22:53:25 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.90.148@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
Jun 20 22:53:25 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996004, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.90.148@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
Jun 20 22:53:27 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:53:27.733174 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39585
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:39585
--ranger_config_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:41371/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:53:27.813405 13560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:53:27.813603 13560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:53:27.813622 13560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:53:27.814759 13560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:53:27.814800 13560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:53:27.814813 13560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:53:27.814826 13560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:53:27.814836 13560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:53:27.816782 13560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.90.148:41371/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:39585
--ranger_config_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:53:27.817003 13560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:53:27.817277 13560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:53:27.822099 13565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:27.822115 13568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:27.822108 13566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:53:27.822304 13560 server_base.cc:1034] running on GCE node
I20260620 22:53:27.822561 13560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:53:27.822930 13560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:53:27.824172 13560 hybrid_clock.cc:648] HybridClock initialized: now 1781996007824160 us; error 28 us; skew 500 ppm
Jun 20 22:53:27 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:53:27.829197 13560 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:53:27.829617 13560 webserver.cc:460] Webserver started at http://127.2.90.190:46851/ using document root <none> and password file <none>
I20260620 22:53:27.829815 13560 fs_manager.cc:359] Metadata directory not provided
I20260620 22:53:27.829852 13560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:53:27.829965 13560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:53:27 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:27.873450 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "0f622a7671654c64adbdccbd4ff270f1"
format_stamp: "Formatted at 2026-06-20 22:53:27 on dist-test-slave-4mtw"
server_key: "a145a64828a0f7721eaeea4f22044819"
server_key_iv: "e3a7eca948d663b3fc06123bc71641ef"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:27.873792 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "0f622a7671654c64adbdccbd4ff270f1"
format_stamp: "Formatted at 2026-06-20 22:53:27 on dist-test-slave-4mtw"
server_key: "a145a64828a0f7721eaeea4f22044819"
server_key_iv: "e3a7eca948d663b3fc06123bc71641ef"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:27.875087 13560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:53:27.886962 13575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:27.887161 13560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:53:27.887248 13560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "0f622a7671654c64adbdccbd4ff270f1"
format_stamp: "Formatted at 2026-06-20 22:53:27 on dist-test-slave-4mtw"
server_key: "a145a64828a0f7721eaeea4f22044819"
server_key_iv: "e3a7eca948d663b3fc06123bc71641ef"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:27.887346 13560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:53:27.899190 13560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:53:27.900923 13560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:53:27.901057 13560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:53:27.905900 13560 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:39585
I20260620 22:53:27.905913 13627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:39585 every 8 connection(s)
I20260620 22:53:27.906277 13560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:53:27.907044 13628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:53:27.908298 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 13560
I20260620 22:53:27.908402 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:53:27.908577 2410 external_mini_cluster.cc:1428] Setting key 8b6f8c62028add583484c065082e6233
I20260620 22:53:27.910274 13628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1: Bootstrap starting.
I20260620 22:53:27.910964 13628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1: Neither blocks nor log segments found. Creating new log.
I20260620 22:53:27.911247 13628 log.cc:824] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:53:27.911962 13628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1: No bootstrap required, opened a new log
Jun 20 22:53:27 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:53:27.914889 13628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f622a7671654c64adbdccbd4ff270f1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39585 } }
I20260620 22:53:27.915127 13628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:53:27.915176 13628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f622a7671654c64adbdccbd4ff270f1, State: Initialized, Role: FOLLOWER
I20260620 22:53:27.915320 13628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f622a7671654c64adbdccbd4ff270f1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39585 } }
I20260620 22:53:27.915397 13628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:53:27.915424 13628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:53:27.915485 13628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:53:27.916077 13628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f622a7671654c64adbdccbd4ff270f1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39585 } }
I20260620 22:53:27.916178 13628 leader_election.cc:302] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f622a7671654c64adbdccbd4ff270f1; no voters:
I20260620 22:53:27.916323 13628 leader_election.cc:288] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:53:27.916368 13633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:53:27.916585 13633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [term 1 LEADER]: Becoming Leader. State: Replica: 0f622a7671654c64adbdccbd4ff270f1, State: Running, Role: LEADER
I20260620 22:53:27.916638 13628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:53:27.916735 13633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f622a7671654c64adbdccbd4ff270f1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39585 } }
I20260620 22:53:27.917069 13633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f622a7671654c64adbdccbd4ff270f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f622a7671654c64adbdccbd4ff270f1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39585 } } }
I20260620 22:53:27.917171 13633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:53:27.917327 13633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f622a7671654c64adbdccbd4ff270f1. Latest consensus state: current_term: 1 leader_uuid: "0f622a7671654c64adbdccbd4ff270f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f622a7671654c64adbdccbd4ff270f1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39585 } } }
I20260620 22:53:27.917424 13633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:53:27.918179 13628 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:53:27.918457 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:27.909067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.2.90.190:39585)
0620 22:53:27.909223 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:27.909233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:27.909264 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:53:27.909835 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:27.909842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:27.909891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:27.910022 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:53:27.910644 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:27.911687 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:27.912478 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:27.912924 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:27.913051 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:27.915093 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:53:27.915097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:53:27.915098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:53:27.915107 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:53:27.916789 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:27.917182 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:27.917185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:27.917185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:27.917276 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:27.917491 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:27.917491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:27.917492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:27.917714 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:53:27.917832 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:27.918161 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 22:53:27.918269 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260620 22:53:27.922606 13639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:53:27.923220 13639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:53:27.924958 13639 catalog_manager.cc:1347] Generated new cluster ID: ab44995c0620413fa48c3a2ed123f856
I20260620 22:53:27.925000 13639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:53:27.937537 13639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:53:27.938309 13639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:53:27.942384 13639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1: Generated new TSK 0
I20260620 22:53:27.942611 13639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:53:28.708730 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:41371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39585
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:53:28.812508 13673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:53:28.812726 13673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:53:28.812758 13673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:53:28.813897 13673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:53:28.813937 13673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:53:28.813969 13673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:53:28.815446 13673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.90.148:41371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:53:28.815673 13673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:53:28.815922 13673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:53:28.822042 13678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:28.822058 13679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:28.822029 13681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:53:28.822762 13673 server_base.cc:1034] running on GCE node
I20260620 22:53:28.822948 13673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:53:28.823191 13673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:53:28.824350 13673 hybrid_clock.cc:648] HybridClock initialized: now 1781996008824337 us; error 27 us; skew 500 ppm
Jun 20 22:53:28 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:53:28.827361 13673 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:53:28.827870 13673 webserver.cc:460] Webserver started at http://127.2.90.129:46759/ using document root <none> and password file <none>
I20260620 22:53:28.828135 13673 fs_manager.cc:359] Metadata directory not provided
I20260620 22:53:28.828205 13673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:53:28.828337 13673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:53:28 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:28.844527 13673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "79eb0325ff884cd3a6aa540958abd7fb"
format_stamp: "Formatted at 2026-06-20 22:53:28 on dist-test-slave-4mtw"
server_key: "a8a8123d7310d3f7c4bc407355876fa5"
server_key_iv: "f329f1db2edee91cef4cc3b548572e3b"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:28.844920 13673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "79eb0325ff884cd3a6aa540958abd7fb"
format_stamp: "Formatted at 2026-06-20 22:53:28 on dist-test-slave-4mtw"
server_key: "a8a8123d7310d3f7c4bc407355876fa5"
server_key_iv: "f329f1db2edee91cef4cc3b548572e3b"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:28.846776 13673 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 22:53:28.867682 13690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:28.867971 13673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 22:53:28.868083 13673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "79eb0325ff884cd3a6aa540958abd7fb"
format_stamp: "Formatted at 2026-06-20 22:53:28 on dist-test-slave-4mtw"
server_key: "a8a8123d7310d3f7c4bc407355876fa5"
server_key_iv: "f329f1db2edee91cef4cc3b548572e3b"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:28.868181 13673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:53:28.887112 13673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:53:28.889506 13673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:53:28.889693 13673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:53:28.889959 13673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:53:28.890360 13673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:53:28.890436 13673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:28.890493 13673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:53:28.890535 13673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:28.897195 13673 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37491
I20260620 22:53:28.897655 13673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:53:28.904398 13803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37491 every 8 connection(s)
I20260620 22:53:28.904873 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 13673
I20260620 22:53:28.904992 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:53:28.905218 2410 external_mini_cluster.cc:1428] Setting key 82823817593af9ddee966a597fad458f
Jun 20 22:53:28 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:53:28.916960 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:28.905296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:50705 (local address 127.2.90.190:39585)
0620 22:53:28.907513 (+ 2217us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:28.907515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:28.907524 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:53:28.907545 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:28.907546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:28.907556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:28.907632 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:53:28.908525 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:28.909360 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:28.910801 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:28.911115 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:28.911167 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:28.913727 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:53:28.913728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:53:28.913729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:53:28.913736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:53:28.915450 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:28.915883 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:28.915884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:28.915885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:28.915951 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:28.916227 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:28.916228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:28.916228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:28.916374 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:53:28.916442 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:28.916778 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 22:53:28.916862 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2191,"thread_start_us":83,"threads_started":1}
I20260620 22:53:28.917184 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:28.905064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39585 (local address 127.2.90.129:50705)
0620 22:53:28.905218 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:28.905268 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:28.905933 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:28.907760 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:28.907765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:28.907990 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:53:28.908425 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:28.908432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:28.909459 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:28.909460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:28.910710 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:28.910713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:28.911230 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:28.911231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:28.911385 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:28.911962 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:53:28.911971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:53:28.913585 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:53:28.915532 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:53:28.915538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:53:28.915540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:53:28.915799 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:53:28.916023 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:53:28.916023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:53:28.916024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:53:28.916177 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:53:28.916440 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:53:28.916445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:53:28.916600 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 22:53:28.916754 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 22:53:28.916899 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
I20260620 22:53:28.917969 13804 heartbeater.cc:347] Connected to a master server at 127.2.90.190:39585
I20260620 22:53:28.918057 13804 heartbeater.cc:464] Registering TS with master...
I20260620 22:53:28.918223 13804 heartbeater.cc:511] Master 127.2.90.190:39585 requested a full tablet report, sending...
I20260620 22:53:28.918740 13592 ts_manager.cc:194] Registered new tserver with Master: 79eb0325ff884cd3a6aa540958abd7fb (127.2.90.129:37491)
I20260620 22:53:28.919605 13592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:50705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:53:28.933554 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:41371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39585
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 22:53:29.059489 13810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:53:29.059827 13810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:53:29.059916 13810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:53:29.061677 13810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:53:29.061736 13810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:53:29.061774 13810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:53:29.063903 13810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.90.148:41371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:53:29.064141 13810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:53:29.064399 13810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:53:29.070236 13818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:29.070243 13817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:29.070495 13820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:53:29.070562 13810 server_base.cc:1034] running on GCE node
I20260620 22:53:29.070766 13810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:53:29.070992 13810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:53:29.072130 13810 hybrid_clock.cc:648] HybridClock initialized: now 1781996009072110 us; error 34 us; skew 500 ppm
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:53:29.075100 13810 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:53:29.075598 13810 webserver.cc:460] Webserver started at http://127.2.90.130:40385/ using document root <none> and password file <none>
I20260620 22:53:29.075834 13810 fs_manager.cc:359] Metadata directory not provided
I20260620 22:53:29.075893 13810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:53:29.076061 13810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:29.091686 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "79426dd49d004800976692113adc5e94"
format_stamp: "Formatted at 2026-06-20 22:53:29 on dist-test-slave-4mtw"
server_key: "98ce248ed4b2caf3f032ef70f367dd53"
server_key_iv: "5daf90248491cf01e29729e2db1eb57c"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:29.092116 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "79426dd49d004800976692113adc5e94"
format_stamp: "Formatted at 2026-06-20 22:53:29 on dist-test-slave-4mtw"
server_key: "98ce248ed4b2caf3f032ef70f367dd53"
server_key_iv: "5daf90248491cf01e29729e2db1eb57c"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:29.094332 13810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:53:29.117321 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:29.117527 13810 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 22:53:29.117626 13810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "79426dd49d004800976692113adc5e94"
format_stamp: "Formatted at 2026-06-20 22:53:29 on dist-test-slave-4mtw"
server_key: "98ce248ed4b2caf3f032ef70f367dd53"
server_key_iv: "5daf90248491cf01e29729e2db1eb57c"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:29.117720 13810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:53:29.136520 13810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:53:29.138859 13810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:53:29.139038 13810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:53:29.139310 13810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:53:29.139689 13810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:53:29.139772 13810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:29.139839 13810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:53:29.139891 13810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:29.147444 13810 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:44959
I20260620 22:53:29.147898 13810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:53:29.156569 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 13810
I20260620 22:53:29.156718 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:53:29.156822 13941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:44959 every 8 connection(s)
I20260620 22:53:29.156905 2410 external_mini_cluster.cc:1428] Setting key b2e40ea4fe98e0d9da18c55ad94df779
I20260620 22:53:29.159197 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.148695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:43003 (local address 127.2.90.190:39585)
0620 22:53:29.148743 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.148745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.148985 (+ 240us) server_negotiation.cc:407] Connection header received
0620 22:53:29.149746 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.149747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.149763 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.149844 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:53:29.150506 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.151274 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.152841 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.153163 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.153234 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.155976 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:53:29.155978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:53:29.155978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:53:29.155985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:53:29.157687 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:29.158149 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:29.158150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:29.158151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:29.158211 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:29.158492 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:29.158492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:29.158493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:29.158620 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:53:29.158658 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.158994 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.159089 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:53:29.160071 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.148664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39585 (local address 127.2.90.130:43003)
0620 22:53:29.148842 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.148888 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.149629 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.149922 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.149926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.150149 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:53:29.150414 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.150421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.151373 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.151374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.152616 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.152619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.153253 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.153255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.153424 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.153968 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:53:29.153978 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:53:29.155555 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:53:29.157771 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:53:29.157777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:53:29.157780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:53:29.158057 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:53:29.158298 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:53:29.158299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:53:29.158299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:53:29.158422 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:53:29.158698 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:53:29.158703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:53:29.158853 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.159629 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.159848 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260620 22:53:29.160655 13942 heartbeater.cc:347] Connected to a master server at 127.2.90.190:39585
I20260620 22:53:29.160775 13942 heartbeater.cc:464] Registering TS with master...
I20260620 22:53:29.160945 13942 heartbeater.cc:511] Master 127.2.90.190:39585 requested a full tablet report, sending...
I20260620 22:53:29.161319 13592 ts_manager.cc:194] Registered new tserver with Master: 79426dd49d004800976692113adc5e94 (127.2.90.130:44959)
I20260620 22:53:29.161800 13592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:43003
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:53:29.188447 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:41371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39585
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:53:29.302474 13947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:53:29.302667 13947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:53:29.302695 13947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:53:29.303793 13947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:53:29.303834 13947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:53:29.303867 13947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:53:29.305286 13947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.90.148:41371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:39585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:53:29.305495 13947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:53:29.305682 13947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:53:29.311702 13955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:29.312202 13952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:53:29.316506 13953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:53:29.317678 13947 server_base.cc:1034] running on GCE node
I20260620 22:53:29.317883 13947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:53:29.318158 13947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:53:29.325528 13947 hybrid_clock.cc:648] HybridClock initialized: now 1781996009325497 us; error 48 us; skew 500 ppm
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:53:29.329737 13947 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:53:29.330265 13947 webserver.cc:460] Webserver started at http://127.2.90.131:42117/ using document root <none> and password file <none>
I20260620 22:53:29.330509 13947 fs_manager.cc:359] Metadata directory not provided
I20260620 22:53:29.330562 13947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:53:29.330680 13947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:29.346405 13947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "d12e5aa497744927a1adc4c2026cfb0c"
format_stamp: "Formatted at 2026-06-20 22:53:29 on dist-test-slave-4mtw"
server_key: "34302fffd8ca29bfe8311603dccdd24e"
server_key_iv: "3152142676a54ced1cb045f8f09d2961"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:29.346840 13947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "d12e5aa497744927a1adc4c2026cfb0c"
format_stamp: "Formatted at 2026-06-20 22:53:29 on dist-test-slave-4mtw"
server_key: "34302fffd8ca29bfe8311603dccdd24e"
server_key_iv: "3152142676a54ced1cb045f8f09d2961"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:29.348508 13947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 22:53:29.363921 13962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:29.364161 13947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 22:53:29.364250 13947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d12e5aa497744927a1adc4c2026cfb0c"
format_stamp: "Formatted at 2026-06-20 22:53:29 on dist-test-slave-4mtw"
server_key: "34302fffd8ca29bfe8311603dccdd24e"
server_key_iv: "3152142676a54ced1cb045f8f09d2961"
server_key_version: "kuduclusterkey@0"
I20260620 22:53:29.364334 13947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:53:29.390465 13947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:53:29.392799 13947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:53:29.392944 13947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:53:29.393162 13947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:53:29.393535 13947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:53:29.393591 13947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:29.393627 13947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:53:29.393647 13947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:53:29.400409 13947 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:43477
I20260620 22:53:29.400830 13947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:53:29.401206 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 13947
I20260620 22:53:29.401322 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:53:29.401539 2410 external_mini_cluster.cc:1428] Setting key 1e1a05d5f2e00395c21b3c29f6e7f864
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:53:29.412587 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.401929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:50483 (local address 127.2.90.190:39585)
0620 22:53:29.402028 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.402031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.402211 (+ 180us) server_negotiation.cc:407] Connection header received
0620 22:53:29.403513 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.403515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.403526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.403553 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:53:29.404343 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.405131 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.406634 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.406926 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.406971 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.409449 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:53:29.409451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:53:29.409451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:53:29.409458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:53:29.411026 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:29.411494 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:29.411495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:29.411496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:29.411555 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:29.411839 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:29.411840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:29.411840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:29.411972 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:53:29.412014 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.412419 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.412494 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 22:53:29.413131 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.401726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:39585 (local address 127.2.90.131:50483)
0620 22:53:29.402098 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.402139 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.403395 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.403734 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.403738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.403963 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:53:29.404229 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.404235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.405241 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.405242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.406537 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.406540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.407059 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.407060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.407203 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.407746 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:53:29.407756 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:53:29.409346 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:53:29.411115 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:53:29.411122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:53:29.411124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:53:29.411405 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:53:29.411621 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:53:29.411622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:53:29.411622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:53:29.411761 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:53:29.412059 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:53:29.412065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:53:29.412228 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.412755 (+ 527us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.412901 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":147,"threads_started":1}
I20260620 22:53:29.413563 14076 heartbeater.cc:347] Connected to a master server at 127.2.90.190:39585
I20260620 22:53:29.413653 14076 heartbeater.cc:464] Registering TS with master...
I20260620 22:53:29.413832 14076 heartbeater.cc:511] Master 127.2.90.190:39585 requested a full tablet report, sending...
I20260620 22:53:29.414173 13592 ts_manager.cc:194] Registered new tserver with Master: d12e5aa497744927a1adc4c2026cfb0c (127.2.90.131:43477)
I20260620 22:53:29.414633 13592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:50483
I20260620 22:53:29.415591 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:43477 every 8 connection(s)
I20260620 22:53:29.423734 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:53:29.432842 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.425297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.2.90.190:39585)
0620 22:53:29.425334 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.425335 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.425606 (+ 271us) server_negotiation.cc:407] Connection header received
0620 22:53:29.425694 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.425694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.425704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.425731 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:53:29.426446 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.427193 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.428241 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.428543 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.428586 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.430009 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:53:29.430010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:53:29.430010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:53:29.430016 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:53:29.431618 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:29.431995 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:29.431996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:29.431996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:29.432049 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:53:29.432265 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:53:29.432273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:53:29.432274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:53:29.432392 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:53:29.432458 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.432658 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.432755 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:53:29.434638 13592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:53:29.436887 13592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:53:29.450169 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.439819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37491 (local address 127.0.0.1:51564)
0620 22:53:29.441309 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.441349 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.441409 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.442241 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.442241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.442242 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.442282 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.442282 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.446127 (+ 3845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.446128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.447258 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.447261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.449884 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.449885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.449964 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.449976 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.450036 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.450084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"thread_start_us":67,"threads_started":1}
I20260620 22:53:29.450583 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.440027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:44959 (local address 127.0.0.1:49456)
0620 22:53:29.441821 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.441830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.441877 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.442121 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.442124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.442126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.442184 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.442186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.444176 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.444177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.447677 (+ 3500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.447680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.450369 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.450370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.450437 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.450440 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.450481 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.450515 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"thread_start_us":159,"threads_started":1}
I20260620 22:53:29.450893 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.440415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.2.90.129:37491)
0620 22:53:29.441529 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.441534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.441543 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:53:29.441575 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.441577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.441593 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.441657 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.442341 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.446037 (+ 3696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.448102 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.448866 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.448925 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.448983 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.450774 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.450818 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":46,"threads_started":1}
I20260620 22:53:29.452042 13728 tablet_service.cc:1467] Processing CreateTablet for tablet 723378ff90094e5e84211ffe5ae04d79 (DEFAULT_TABLE table=test-table [id=650dad5e88e94df58650f16569d51355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:53:29.452445 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723378ff90094e5e84211ffe5ae04d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:53:29.456359 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.440537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.2.90.130:44959)
0620 22:53:29.441772 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.441776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.441923 (+ 147us) server_negotiation.cc:407] Connection header received
0620 22:53:29.441959 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.441961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.441975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.442040 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.443105 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.444089 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.447779 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.449717 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.449782 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.449846 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.456216 (+ 6370us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.456277 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":75,"threads_started":1}
I20260620 22:53:29.457037 13876 tablet_service.cc:1467] Processing CreateTablet for tablet 723378ff90094e5e84211ffe5ae04d79 (DEFAULT_TABLE table=test-table [id=650dad5e88e94df58650f16569d51355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:53:29.457382 13876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723378ff90094e5e84211ffe5ae04d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:53:29.460553 14095 tablet_bootstrap.cc:492] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: Bootstrap starting.
I20260620 22:53:29.461323 14095 tablet_bootstrap.cc:654] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: Neither blocks nor log segments found. Creating new log.
I20260620 22:53:29.461620 14095 log.cc:824] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: Log is configured to *not* fsync() on all Append() calls
I20260620 22:53:29.462375 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.440849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:43477 (local address 127.0.0.1:59340)
0620 22:53:29.445395 (+ 4546us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.445405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.445449 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.456901 (+ 11452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.456903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.456905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.456947 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.456948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.458281 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.458282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.459357 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.459359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.462150 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.462151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.462217 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.462222 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.462250 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.462301 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4531,"thread_start_us":62,"threads_started":1}
I20260620 22:53:29.462919 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.441103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.2.90.131:43477)
0620 22:53:29.456728 (+ 15625us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.456734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.456745 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:53:29.456781 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.456783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.456798 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.457013 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.457087 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.458173 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.459472 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.462055 (+ 2583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.462695 (+ 640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.462778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.462804 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.462843 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15564,"thread_start_us":80,"threads_started":1}
I20260620 22:53:29.463627 14010 tablet_service.cc:1467] Processing CreateTablet for tablet 723378ff90094e5e84211ffe5ae04d79 (DEFAULT_TABLE table=test-table [id=650dad5e88e94df58650f16569d51355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:53:29.463937 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723378ff90094e5e84211ffe5ae04d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:53:29.483112 14098 tablet_bootstrap.cc:492] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Bootstrap starting.
I20260620 22:53:29.488227 14095 tablet_bootstrap.cc:492] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: No bootstrap required, opened a new log
I20260620 22:53:29.488405 14095 ts_tablet_manager.cc:1397] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20260620 22:53:29.491384 14095 raft_consensus.cc:348] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.491701 14095 raft_consensus.cc:374] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:53:29.491797 14095 raft_consensus.cc:729] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79eb0325ff884cd3a6aa540958abd7fb, State: Initialized, Role: FOLLOWER
I20260620 22:53:29.492012 14095 consensus_queue.cc:260] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.492432 14095 ts_tablet_manager.cc:1428] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 22:53:29.492617 13804 heartbeater.cc:503] Master 127.2.90.190:39585 was elected leader, sending a full tablet report...
I20260620 22:53:29.493675 14098 tablet_bootstrap.cc:654] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Neither blocks nor log segments found. Creating new log.
I20260620 22:53:29.494086 14098 log.cc:824] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:53:29.494349 14096 tablet_bootstrap.cc:492] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Bootstrap starting.
I20260620 22:53:29.495683 14096 tablet_bootstrap.cc:654] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Neither blocks nor log segments found. Creating new log.
I20260620 22:53:29.497711 14096 log.cc:824] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Log is configured to *not* fsync() on all Append() calls
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:29.506263 14098 tablet_bootstrap.cc:492] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: No bootstrap required, opened a new log
I20260620 22:53:29.506361 14098 ts_tablet_manager.cc:1397] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20260620 22:53:29.509573 14098 raft_consensus.cc:348] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.509770 14098 raft_consensus.cc:374] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:53:29.509811 14098 raft_consensus.cc:729] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12e5aa497744927a1adc4c2026cfb0c, State: Initialized, Role: FOLLOWER
I20260620 22:53:29.509949 14098 consensus_queue.cc:260] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.510251 14098 ts_tablet_manager.cc:1428] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 22:53:29.510401 14076 heartbeater.cc:503] Master 127.2.90.190:39585 was elected leader, sending a full tablet report...
I20260620 22:53:29.511636 14096 tablet_bootstrap.cc:492] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: No bootstrap required, opened a new log
I20260620 22:53:29.511727 14096 ts_tablet_manager.cc:1397] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260620 22:53:29.513746 14096 raft_consensus.cc:348] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.513916 14096 raft_consensus.cc:374] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:53:29.513953 14096 raft_consensus.cc:729] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79426dd49d004800976692113adc5e94, State: Initialized, Role: FOLLOWER
I20260620 22:53:29.514073 14096 consensus_queue.cc:260] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.514331 14096 ts_tablet_manager.cc:1428] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:53:29.514437 13942 heartbeater.cc:503] Master 127.2.90.190:39585 was elected leader, sending a full tablet report...
W20260620 22:53:29.648871 13943 tablet.cc:2369] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:53:29.655382 13805 tablet.cc:2369] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:53:29.664605 14077 tablet.cc:2369] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 22:53:29 dist-test-slave-4mtw krb5kdc[13116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996009, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:53:29.798090 14102 raft_consensus.cc:482] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:53:29.798238 14102 raft_consensus.cc:504] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.798686 14102 leader_election.cc:288] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79eb0325ff884cd3a6aa540958abd7fb (127.2.90.129:37491), 79426dd49d004800976692113adc5e94 (127.2.90.130:44959)
I20260620 22:53:29.803081 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.799101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37553 (local address 127.2.90.130:44959)
0620 22:53:29.799163 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.799165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.799507 (+ 342us) server_negotiation.cc:407] Connection header received
0620 22:53:29.799598 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.799599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.799608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.799633 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.799825 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.800537 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.801821 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.802371 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.802417 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.802426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.802970 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.803014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:53:29.803184 14103 raft_consensus.cc:482] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:53:29.803257 14103 raft_consensus.cc:504] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.803567 14103 leader_election.cc:288] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d12e5aa497744927a1adc4c2026cfb0c (127.2.90.131:43477), 79eb0325ff884cd3a6aa540958abd7fb (127.2.90.129:37491)
I20260620 22:53:29.804036 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.798849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37491 (local address 127.2.90.131:41905)
0620 22:53:29.798891 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.798904 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.798960 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.799316 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.799317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.799319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.799492 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.799495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.800356 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.800357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.801521 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.801524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.803791 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.803792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.803873 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.803876 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.803920 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.803960 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 22:53:29.804291 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.799112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:41905 (local address 127.2.90.129:37491)
0620 22:53:29.799193 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.799194 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.799201 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:53:29.799229 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.799229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.799239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.799297 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.799587 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.800266 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.803123 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.803693 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.803753 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.803762 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.804187 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.804230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 22:53:29.804737 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723378ff90094e5e84211ffe5ae04d79" candidate_uuid: "d12e5aa497744927a1adc4c2026cfb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79eb0325ff884cd3a6aa540958abd7fb" is_pre_election: true
I20260620 22:53:29.804872 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.798896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:44959 (local address 127.2.90.131:37553)
0620 22:53:29.799424 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.799435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.799532 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.799687 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.799688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.799689 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.799764 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.799765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.800634 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.800635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.801727 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.801730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.802812 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.802813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.802881 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.802896 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.804743 (+ 1847us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.804788 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":86,"threads_started":1}
I20260620 22:53:29.804890 13758 raft_consensus.cc:2454] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12e5aa497744927a1adc4c2026cfb0c in term 0.
I20260620 22:53:29.805135 13896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723378ff90094e5e84211ffe5ae04d79" candidate_uuid: "d12e5aa497744927a1adc4c2026cfb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79426dd49d004800976692113adc5e94" is_pre_election: true
I20260620 22:53:29.805229 13896 raft_consensus.cc:2454] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12e5aa497744927a1adc4c2026cfb0c in term 0.
I20260620 22:53:29.805212 13965 leader_election.cc:302] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79eb0325ff884cd3a6aa540958abd7fb, d12e5aa497744927a1adc4c2026cfb0c; no voters:
I20260620 22:53:29.805363 14102 raft_consensus.cc:2781] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:53:29.805425 14102 raft_consensus.cc:482] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:53:29.805445 14102 raft_consensus.cc:3037] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:53:29.806264 14102 raft_consensus.cc:504] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.806372 14102 leader_election.cc:288] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [CANDIDATE]: Term 1 election: Requested vote from peers 79eb0325ff884cd3a6aa540958abd7fb (127.2.90.129:37491), 79426dd49d004800976692113adc5e94 (127.2.90.130:44959)
I20260620 22:53:29.806613 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723378ff90094e5e84211ffe5ae04d79" candidate_uuid: "d12e5aa497744927a1adc4c2026cfb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79eb0325ff884cd3a6aa540958abd7fb"
I20260620 22:53:29.806679 13758 raft_consensus.cc:3037] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:53:29.807442 13758 raft_consensus.cc:2454] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12e5aa497744927a1adc4c2026cfb0c in term 1.
I20260620 22:53:29.807586 13965 leader_election.cc:302] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79eb0325ff884cd3a6aa540958abd7fb, d12e5aa497744927a1adc4c2026cfb0c; no voters:
I20260620 22:53:29.807701 13896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723378ff90094e5e84211ffe5ae04d79" candidate_uuid: "d12e5aa497744927a1adc4c2026cfb0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79426dd49d004800976692113adc5e94"
I20260620 22:53:29.807757 13896 raft_consensus.cc:3037] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:53:29.807787 14102 raft_consensus.cc:2781] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:53:29.807953 14102 raft_consensus.cc:686] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 1 LEADER]: Becoming Leader. State: Replica: d12e5aa497744927a1adc4c2026cfb0c, State: Running, Role: LEADER
I20260620 22:53:29.808039 14102 consensus_queue.cc:237] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } }
I20260620 22:53:29.808480 13896 raft_consensus.cc:2454] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12e5aa497744927a1adc4c2026cfb0c in term 1.
I20260620 22:53:29.809197 13592 catalog_manager.cc:5495] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c reported cstate change: term changed from 0 to 1, leader changed from <none> to d12e5aa497744927a1adc4c2026cfb0c (127.2.90.131). New cstate: current_term: 1 leader_uuid: "d12e5aa497744927a1adc4c2026cfb0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12e5aa497744927a1adc4c2026cfb0c" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 43477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 } health_report { overall_health: UNKNOWN } } }
I20260620 22:53:29.813217 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.804259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:43477 (local address 127.2.90.130:41163)
0620 22:53:29.808630 (+ 4371us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.808639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.808705 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.809904 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.809905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.809907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.810046 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.810048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.811087 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.811088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.812182 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.812185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.812953 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.812954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.813022 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.813062 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.813104 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.813141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4358,"mutex_wait_us":33,"thread_start_us":37,"threads_started":1}
I20260620 22:53:29.813915 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.804365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:41163 (local address 127.2.90.131:43477)
0620 22:53:29.804397 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.804399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.808709 (+ 4310us) server_negotiation.cc:407] Connection header received
0620 22:53:29.808803 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.808804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.808813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.808853 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.810123 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.810780 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.812330 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.812845 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.812914 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.812922 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.813815 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.813854 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:53:29.815291 14030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723378ff90094e5e84211ffe5ae04d79" candidate_uuid: "79426dd49d004800976692113adc5e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12e5aa497744927a1adc4c2026cfb0c" is_pre_election: true
I20260620 22:53:29.816047 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.803788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37491 (local address 127.2.90.130:49743)
0620 22:53:29.805222 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.805232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:53:29.808493 (+ 3261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:53:29.811918 (+ 3425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:53:29.811920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:53:29.811922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:53:29.812004 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.812005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.812892 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.812893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.814474 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:53:29.814477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.815802 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.815803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:53:29.815860 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.815864 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:53:29.815923 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:53:29.815956 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"thread_start_us":76,"threads_started":1}
I20260620 22:53:29.816416 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.804075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:49743 (local address 127.2.90.129:37491)
0620 22:53:29.804361 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.804363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.805281 (+ 918us) server_negotiation.cc:407] Connection header received
0620 22:53:29.811819 (+ 6538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.811820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.811829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.811881 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:53:29.812074 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.812808 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.814560 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.815070 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.815127 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.815134 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.816313 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.816355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":76,"threads_started":1}
I20260620 22:53:29.816663 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723378ff90094e5e84211ffe5ae04d79" candidate_uuid: "79426dd49d004800976692113adc5e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79eb0325ff884cd3a6aa540958abd7fb" is_pre_election: true
I20260620 22:53:29.816781 13758 raft_consensus.cc:2379] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79426dd49d004800976692113adc5e94 in current term 1: Already voted for candidate d12e5aa497744927a1adc4c2026cfb0c in this term.
I20260620 22:53:29.817003 13831 leader_election.cc:302] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79426dd49d004800976692113adc5e94; no voters: 79eb0325ff884cd3a6aa540958abd7fb, d12e5aa497744927a1adc4c2026cfb0c
I20260620 22:53:29.817171 14103 raft_consensus.cc:2726] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 22:53:29.895216 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.891900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.2.90.131:43477)
0620 22:53:29.891940 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:53:29.891942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:53:29.891950 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:53:29.891992 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:53:29.891992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:53:29.892016 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:53:29.892090 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:53:29.892254 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.893022 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.894136 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:53:29.894440 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:53:29.894509 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:53:29.894686 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:53:29.894822 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:53:29.894906 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:53:29.895117 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 22:53:29.895137 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:53:29.897176 13896 raft_consensus.cc:1261] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 1 FOLLOWER]: Refusing update from remote peer d12e5aa497744927a1adc4c2026cfb0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:53:29.897447 13758 raft_consensus.cc:1261] T 723378ff90094e5e84211ffe5ae04d79 P 79eb0325ff884cd3a6aa540958abd7fb [term 1 FOLLOWER]: Refusing update from remote peer d12e5aa497744927a1adc4c2026cfb0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:53:29.897838 14110 consensus_queue.cc:1035] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79eb0325ff884cd3a6aa540958abd7fb" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:53:29.898070 14102 consensus_queue.cc:1035] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "79426dd49d004800976692113adc5e94" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:53:29.901999 14114 mvcc.cc:205] Tried to move back new op lower bound from 7299055656535732224 to 7299055656174088192. Current Snapshot: MvccSnapshot[applied={T|T < 7299055656535732224}]
I20260620 22:53:29.902351 14113 mvcc.cc:205] Tried to move back new op lower bound from 7299055656535732224 to 7299055656174088192. Current Snapshot: MvccSnapshot[applied={T|T < 7299055656535732224}]
I20260620 22:53:29.903407 14115 mvcc.cc:205] Tried to move back new op lower bound from 7299055656535732224 to 7299055656174088192. Current Snapshot: MvccSnapshot[applied={T|T < 7299055656535732224}]
I20260620 22:53:29.904961 13591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:53:29.905050 13591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:53:29.905854 13591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f622a7671654c64adbdccbd4ff270f1: Sending DeleteTablet for 3 replicas of tablet 723378ff90094e5e84211ffe5ae04d79
I20260620 22:53:29.906244 14010 tablet_service.cc:1514] Processing DeleteTablet for tablet 723378ff90094e5e84211ffe5ae04d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:53:29 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:59340
I20260620 22:53:29.906525 13728 tablet_service.cc:1514] Processing DeleteTablet for tablet 723378ff90094e5e84211ffe5ae04d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:53:29 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:51564
I20260620 22:53:29.906765 13876 tablet_service.cc:1514] Processing DeleteTablet for tablet 723378ff90094e5e84211ffe5ae04d79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:53:29 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:49456
I20260620 22:53:29.907322 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 13673
I20260620 22:53:29.908033 14122 tablet_replica.cc:330] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: stopping tablet replica
I20260620 22:53:29.908115 14122 raft_consensus.cc:2229] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:53:29.908215 14122 raft_consensus.cc:2258] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:53:29.908329 14123 tablet_replica.cc:330] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: stopping tablet replica
I20260620 22:53:29.908401 14123 raft_consensus.cc:2229] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:53:29.908478 14123 raft_consensus.cc:2258] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:53:29.908962 14123 ts_tablet_manager.cc:1905] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:53:29.911183 14122 ts_tablet_manager.cc:1905] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:53:29.911321 14123 ts_tablet_manager.cc:1918] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:53:29.911439 14123 log.cc:1196] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/723378ff90094e5e84211ffe5ae04d79
I20260620 22:53:29.912043 14123 ts_tablet_manager.cc:1939] T 723378ff90094e5e84211ffe5ae04d79 P 79426dd49d004800976692113adc5e94: Deleting consensus metadata
I20260620 22:53:29.912521 13576 catalog_manager.cc:4841] TS 79426dd49d004800976692113adc5e94 (127.2.90.130:44959): tablet 723378ff90094e5e84211ffe5ae04d79 (table test-table [id=650dad5e88e94df58650f16569d51355]) successfully deleted
I20260620 22:53:29.912611 14122 ts_tablet_manager.cc:1918] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:53:29.912672 14122 log.cc:1196] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/723378ff90094e5e84211ffe5ae04d79
I20260620 22:53:29.912925 14122 ts_tablet_manager.cc:1939] T 723378ff90094e5e84211ffe5ae04d79 P d12e5aa497744927a1adc4c2026cfb0c: Deleting consensus metadata
I20260620 22:53:29.913308 13579 catalog_manager.cc:4841] TS d12e5aa497744927a1adc4c2026cfb0c (127.2.90.131:43477): tablet 723378ff90094e5e84211ffe5ae04d79 (table test-table [id=650dad5e88e94df58650f16569d51355]) successfully deleted
W20260620 22:53:29.921577 13578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:37491 (error 108)
I20260620 22:53:29.922219 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 13810
I20260620 22:53:29.922811 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:53:29.922367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37491 (local address 127.0.0.1:51580)
0620 22:53:29.922728 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 22:53:29.922776 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
W20260620 22:53:29.923390 13578 catalog_manager.cc:4584] TS 79eb0325ff884cd3a6aa540958abd7fb (127.2.90.129:37491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 723378ff90094e5e84211ffe5ae04d79: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37491: connect: Connection refused (error 111)
I20260620 22:53:29.927213 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 13947
I20260620 22:53:29.933265 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 13560
I20260620 22:53:29.937999 2410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:53:30.271509 2410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:53:30.271610 2410 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:53:30.605435 2410 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:53:30.605 UTC [13146] LOG: received smart shutdown request
2026-06-20 22:53:30.608 UTC [13146] LOG: background worker "logical replication launcher" (PID 13155) exited with exit code 1
2026-06-20 22:53:30.609 UTC [13150] LOG: shutting down
2026-06-20 22:53:30.739 UTC [13146] LOG: database system is shut down
2026-06-20T22:53:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43560 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:53:30 dist-test-slave-4mtw krb5kdc[14134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:53:30 dist-test-slave-4mtw krb5kdc[14134](info): set up 2 sockets
Jun 20 22:53:30 dist-test-slave-4mtw krb5kdc[14134](info): commencing operation
krb5kdc: starting...
W20260620 22:53:31.772382 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:53:31 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:53:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/rangeradmin_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/rangeradmin_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/rangerlookup_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/rangerlookup_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
I20260620 22:53:31.917336 2410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5TG4FZ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/postgres -l logfile start
2026-06-20 22:53:34.071 UTC [14164] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 22:53:34.071 UTC [14164] LOG: listening on IPv4 address "127.2.90.148", port 33097
2026-06-20 22:53:34.074 UTC [14164] LOG: listening on Unix socket "/tmp/.s.PGSQL.33097"
2026-06-20 22:53:34.099 UTC [14167] LOG: database system was shut down at 2026-06-20 22:53:32 UTC
2026-06-20 22:53:34.103 UTC [14164] LOG: database system is ready to accept connections
I20260620 22:53:34.976251 2410 mini_postgres.cc:96] Postgres bound to 33097
2026-06-20 22:53:34.981 UTC [14175] FATAL: database "slave" does not exist
127.2.90.148:33097 - accepting connections
I20260620 22:53:34.982105 2410 mini_ranger.cc:161] Starting Ranger...
I20260620 22:53:34.995479 2410 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 22:53:35.380617 2410 mini_ranger.cc:87] Created ranger Postgres database
I20260620 22:53:35.380713 2410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-admin
2026-06-20 22:53:35,642 [I] DB FLAVOR :POSTGRES
2026-06-20 22:53:35,642 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:53:35,642 [I] Checking connection..
2026-06-20 22:53:35,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 22:53:35,893 [I] Checking connection passed.
2026-06-20 22:53:35,893 [I] --------- Verifying version history table ---------
2026-06-20 22:53:35,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:53:36,144 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:53:36,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:53:36,372 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 22:53:36,372 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 22:53:36,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 22:53:36.602 UTC [14239] WARNING: there is no transaction in progress
2026-06-20 22:53:36,608 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 22:53:36,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 22:53:36,857 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 22:53:36,858 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:53:36,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 22:53:37,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 22:53:37,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4mtw.c.gcp-upstream.internal','N') ;"
2026-06-20 22:53:37,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:53:37,801 [I] Table x_portal_user does not exist in database ranger
2026-06-20 22:53:37,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:53:38,045 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 22:53:38,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:53:38,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:53:38,492 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 22:53:38,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 22:53:39.346 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.361 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.376 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.389 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.403 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.421 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.471 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.477 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.488 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.497 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.506 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.518 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.526 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.531 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.536 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:39.541 UTC [14347] WARNING: there is no transaction in progress
2026-06-20 22:53:40,060 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 22:53:40,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 22:53:40,292 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 22:53:40,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 22:53:40,529 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 22:53:40,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 22:53:40,747 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 22:53:40,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 22:53:40,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 22:53:41,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4mtw.c.gcp-upstream.internal';"
2026-06-20 22:53:41,458 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 22:53:41,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 22:53:41,698 [I] DB_PATCHES have already been applied
I20260620 22:53:41.703800 2410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-admin:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5TG4FZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 22:53:41.703864 2410 mini_ranger.cc:193] Using host: 127.2.90.148
I20260620 22:53:41.706615 2410 mini_ranger.cc:227] Ranger admin URL: http://127.2.90.148:50545
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 10:53:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 22:53:42.734190 2410 mini_ranger.cc:160] Time spent starting Ranger: real 7.752s user 0.000s sys 0.008s
Jun 20, 2026 10:54:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 10:54:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 10:54:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 10:54:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 10:54:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 22:54:03.462373 2410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.90.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.90.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/rangerkms_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.90.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/rangerkms_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.90.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.90.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.90.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/HTTP_127.2.90.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 22:54:03 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996043, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 22:54:03.531083 2410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 22:54:03.538916 2410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 22:54:04.225373 2410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 22:54:04.225449 2410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms
I20260620 22:54:04.225464 2410 mini_ranger_kms.cc:227] Using postgres at 127.2.90.148:33097
2026-06-20 22:54:04,366 [I] DB FLAVOR :POSTGRES
2026-06-20 22:54:04,366 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 22:54:04,366 [I] Checking connection
2026-06-20 22:54:04,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 22:54:04,583 [I] connection success
2026-06-20 22:54:04,584 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 22:54:04,584 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 22:54:04,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 22:54:04,812 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 22:54:04,812 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 22:54:04,813 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 22:54:04,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.90.148:33097/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 22:54:05,058 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 22:54:05.493198 2410 mini_ranger_kms.cc:310] Created kms service
I20260620 22:54:05.664942 2410 mini_ranger_kms.cc:326] Created kudu user
I20260620 22:54:05.704298 2410 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 22:54:05 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996043, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:06.048229 2410 mini_ranger_kms.cc:384] Added ranger policy
I20260620 22:54:06.048341 2410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms:/tmp/dist-test-task5TG4FZ/build/release/bin/postgresql.jar:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5TG4FZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 22:54:06.048360 2410 mini_ranger_kms.cc:240] Using host: 127.2.90.148
I20260620 22:54:06.051481 2410 mini_ranger_kms.cc:276] Ranger KMS PID: 14527
I20260620 22:54:06.051568 2410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.90.148:56361
Jun 20, 2026 10:54:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 10:54:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 10:54:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 10:54:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 10:54:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 10:54:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 22:54:08.015789 2410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.485s user 0.006s sys 0.117s
I20260620 22:54:08.015892 2410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 22:54:08.015937 2410 mini_ranger_kms.cc:401] 127.2.90.148:56361/kms/v1/keys
Jun 20 22:54:08 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.90.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 22:54:08 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.90.148@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
Jun 20 22:54:08 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.90.148@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
Jun 20 22:54:11 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:54:11.157001 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45573
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:45573
--ranger_config_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:11.237175 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:11.237375 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:11.237394 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:11.238499 14578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:11.238539 14578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:11.238551 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:11.238574 14578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:11.238597 14578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:11.240136 14578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.90.148:56361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:45573
--ranger_config_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:11.240360 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:11.240557 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:11.245189 14583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:11.245198 14584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:11.245199 14586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:11.245419 14578 server_base.cc:1034] running on GCE node
I20260620 22:54:11.245618 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:11.245885 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:11.247021 14578 hybrid_clock.cc:648] HybridClock initialized: now 1781996051247005 us; error 27 us; skew 500 ppm
Jun 20 22:54:11 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:11.252508 14578 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:11.252960 14578 webserver.cc:460] Webserver started at http://127.2.90.190:35823/ using document root <none> and password file <none>
I20260620 22:54:11.253142 14578 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:11.253176 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:11.253268 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:54:11 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:11.304721 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "3b605533cde044eaa8702b2535620999"
format_stamp: "Formatted at 2026-06-20 22:54:11 on dist-test-slave-4mtw"
server_key: "48bafec7df1fff354615af57fa1a43ad"
server_key_iv: "35a2b90abc374431722218904de30d07"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:11.305120 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "3b605533cde044eaa8702b2535620999"
format_stamp: "Formatted at 2026-06-20 22:54:11 on dist-test-slave-4mtw"
server_key: "48bafec7df1fff354615af57fa1a43ad"
server_key_iv: "35a2b90abc374431722218904de30d07"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:11.306465 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:54:11.319962 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:11.320125 14578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:11.320209 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "3b605533cde044eaa8702b2535620999"
format_stamp: "Formatted at 2026-06-20 22:54:11 on dist-test-slave-4mtw"
server_key: "48bafec7df1fff354615af57fa1a43ad"
server_key_iv: "35a2b90abc374431722218904de30d07"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:11.320294 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:11.346825 14578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:11.348671 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:11.348851 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:11.353644 14578 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:45573
I20260620 22:54:11.353706 14645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:45573 every 8 connection(s)
I20260620 22:54:11.354066 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:11.354727 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:11.357040 14646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Bootstrap starting.
I20260620 22:54:11.357597 14646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:11.357851 14646 log.cc:824] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:11.358506 14646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: No bootstrap required, opened a new log
I20260620 22:54:11.360406 14646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:11.360555 14646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:11.360587 14646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b605533cde044eaa8702b2535620999, State: Initialized, Role: FOLLOWER
I20260620 22:54:11.360757 14646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:11.360836 14646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:11.360868 14646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:11.360915 14646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:11.361418 14646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:11.361511 14646 leader_election.cc:302] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b605533cde044eaa8702b2535620999; no voters:
I20260620 22:54:11.361672 14646 leader_election.cc:288] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:54:11.361708 14649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:11.361811 14649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 LEADER]: Becoming Leader. State: Replica: 3b605533cde044eaa8702b2535620999, State: Running, Role: LEADER
I20260620 22:54:11.361924 14646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:11.361955 14649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:11.362393 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b605533cde044eaa8702b2535620999. Latest consensus state: current_term: 1 leader_uuid: "3b605533cde044eaa8702b2535620999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } } }
I20260620 22:54:11.362406 14650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b605533cde044eaa8702b2535620999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } } }
I20260620 22:54:11.362494 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:11.362504 14650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:11.362627 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 14578
I20260620 22:54:11.362715 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:54:11.362742 14655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:11.362895 2410 external_mini_cluster.cc:1428] Setting key 6290d4edf535d51f6c3f857dd0306987
I20260620 22:54:11.363225 14655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:11.364042 14646 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:54:11.377255 14655 catalog_manager.cc:1347] Generated new cluster ID: c038911e03ca490e9df1929c0872ebe9
I20260620 22:54:11.377324 14655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:54:11 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:11.381870 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:11.363425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.2.90.190:45573)
0620 22:54:11.364166 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:11.364177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:11.364564 (+ 387us) server_negotiation.cc:407] Connection header received
0620 22:54:11.366137 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:11.366185 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:11.366360 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:11.366573 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:11.367750 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:11.369192 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:11.370243 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:11.370858 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:11.370965 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:11.378254 (+ 7289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:11.378258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:11.378259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:11.378271 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:11.380065 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:11.380559 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:11.380563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:11.380563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:11.380663 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:11.380978 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:11.380979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:11.380979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:11.381290 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:11.381414 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:11.381541 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 22:54:11.381656 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":72,"threads_started":1}
I20260620 22:54:11.384449 14655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:54:11.385203 14655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:11.392520 14655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Generated new TSK 0
I20260620 22:54:11.392751 14655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:54:12.064952 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45573
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:12.163364 14691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:12.163551 14691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:12.163579 14691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:12.164755 14691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:12.164794 14691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:12.164829 14691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:12.166265 14691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.90.148:56361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:12.166468 14691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:12.166683 14691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:12.171295 14697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.171301 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.171679 14696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:12.171713 14691 server_base.cc:1034] running on GCE node
I20260620 22:54:12.171909 14691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:12.172176 14691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:12.173336 14691 hybrid_clock.cc:648] HybridClock initialized: now 1781996052173330 us; error 38 us; skew 500 ppm
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:12.176192 14691 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:12.176744 14691 webserver.cc:460] Webserver started at http://127.2.90.129:33359/ using document root <none> and password file <none>
I20260620 22:54:12.177007 14691 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:12.177079 14691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:12.177201 14691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:12.189500 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "6593658f77bd4d5b86fcd0996930bcc2"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "b4b8ea80ae249ff919750b01c62dec2f"
server_key_iv: "40e98000f4cf45e837f3e8a0473320cf"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.189994 14691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "6593658f77bd4d5b86fcd0996930bcc2"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "b4b8ea80ae249ff919750b01c62dec2f"
server_key_iv: "40e98000f4cf45e837f3e8a0473320cf"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.191933 14691 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:54:12.205078 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.205291 14691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:12.205406 14691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "6593658f77bd4d5b86fcd0996930bcc2"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "b4b8ea80ae249ff919750b01c62dec2f"
server_key_iv: "40e98000f4cf45e837f3e8a0473320cf"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.205505 14691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:12.231527 14691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:12.233866 14691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:12.234038 14691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:12.234277 14691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:12.234637 14691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:12.234675 14691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.234696 14691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:12.234707 14691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.241153 14691 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:46755
I20260620 22:54:12.241263 14821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:46755 every 8 connection(s)
I20260620 22:54:12.241739 14691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:12.251907 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:12.242518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:34279 (local address 127.2.90.190:45573)
0620 22:54:12.242657 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:12.242659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:12.242729 (+ 70us) server_negotiation.cc:407] Connection header received
0620 22:54:12.243408 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:12.243410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:12.243425 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:12.243513 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:12.244208 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.245067 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.246528 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.246847 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.246914 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:12.249305 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:12.249307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:12.249307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:12.249315 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:12.250660 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:12.251030 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:12.251031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:12.251031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:12.251093 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:12.251302 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:12.251303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:12.251303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:12.251423 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:12.251476 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:12.251726 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 22:54:12.251808 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260620 22:54:12.251976 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 14691
I20260620 22:54:12.252055 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:12.242464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45573 (local address 127.2.90.129:34279)
0620 22:54:12.242612 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:12.242654 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:12.243279 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:12.243594 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:12.243598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:12.243828 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:12.244108 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:12.244114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.245159 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.245160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:12.246440 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:12.246443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.246948 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.246949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:12.247088 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:12.247670 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:12.247680 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:12.249215 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:12.250732 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:12.250738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:12.250740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:12.250965 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:12.251146 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:12.251147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:12.251147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:12.251249 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:12.251477 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:12.251483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:12.251612 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 22:54:12.251698 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:54:12.251842 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260620 22:54:12.252064 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:54:12.252259 2410 external_mini_cluster.cc:1428] Setting key 9e92c0aa840eb5d3335f212bec07c605
I20260620 22:54:12.252727 14822 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45573
I20260620 22:54:12.252822 14822 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:12.253001 14822 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:12.253546 14610 ts_manager.cc:194] Registered new tserver with Master: 6593658f77bd4d5b86fcd0996930bcc2 (127.2.90.129:46755)
I20260620 22:54:12.254357 14610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:34279
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:54:12.277311 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45573
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:12.383752 14828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:12.383946 14828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:12.383975 14828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:12.385241 14828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:12.385281 14828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:12.385314 14828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:54:12.386794 14828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.90.148:56361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:12.387020 14828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:12.387244 14828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 22:54:12.392063 14838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.392153 14836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.392280 14835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:12.392750 14828 server_base.cc:1034] running on GCE node
I20260620 22:54:12.392941 14828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:12.393165 14828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:12.394310 14828 hybrid_clock.cc:648] HybridClock initialized: now 1781996052394290 us; error 33 us; skew 500 ppm
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:12.396926 14828 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:54:12.397347 14828 webserver.cc:460] Webserver started at http://127.2.90.130:44793/ using document root <none> and password file <none>
I20260620 22:54:12.397570 14828 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:12.397619 14828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:12.397770 14828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:12.413087 14828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "0b313404234d4884ab70d24297f36a7d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "d87e9b3d704cd7a41c52db3fbabaa523"
server_key_iv: "4f805d56414d78808ec132ae89119771"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.413546 14828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "0b313404234d4884ab70d24297f36a7d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "d87e9b3d704cd7a41c52db3fbabaa523"
server_key_iv: "4f805d56414d78808ec132ae89119771"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.419162 14828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 22:54:12.443209 14845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.443471 14828 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.000s
I20260620 22:54:12.443552 14828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "0b313404234d4884ab70d24297f36a7d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "d87e9b3d704cd7a41c52db3fbabaa523"
server_key_iv: "4f805d56414d78808ec132ae89119771"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.443640 14828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:12.479851 14828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:12.482358 14828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:12.482550 14828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:12.482812 14828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:12.483210 14828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:12.483287 14828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.483367 14828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:12.483420 14828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.490830 14828 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45819
I20260620 22:54:12.491281 14828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:54:12.493259 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45819 every 8 connection(s)
I20260620 22:54:12.500162 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 14828
I20260620 22:54:12.500253 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:54:12.500448 2410 external_mini_cluster.cc:1428] Setting key f254b1175a66fd8e3678f11590908f09
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:12.515829 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:12.497576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:59571 (local address 127.2.90.190:45573)
0620 22:54:12.497624 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:12.497626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:12.504992 (+ 7366us) server_negotiation.cc:407] Connection header received
0620 22:54:12.505742 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:12.505744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:12.505761 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:12.505792 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:12.506704 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.507452 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.509516 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.509808 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.509856 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:12.512419 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:12.512420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:12.512421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:12.512428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:12.514056 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:12.514542 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:12.514543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:12.514544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:12.514599 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:12.515074 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:12.515075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:12.515075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:12.515209 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:12.515255 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:12.515642 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 22:54:12.515731 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:54:12.516570 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:12.497285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45573 (local address 127.2.90.130:59571)
0620 22:54:12.504831 (+ 7546us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:12.504895 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:12.505607 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:12.505997 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:12.506002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:12.506235 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:12.506593 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:12.506600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.508058 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.508058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:12.509415 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:12.509418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.509960 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.509960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:12.510100 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:12.510669 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:12.510679 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:12.512300 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:12.514144 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:12.514150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:12.514153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:12.514456 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:12.514836 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:12.514837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:12.514837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:12.514986 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:12.515303 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:12.515310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:12.515480 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 22:54:12.516168 (+ 688us) client_negotiation.cc:240] Negotiation successful
0620 22:54:12.516337 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7487,"thread_start_us":93,"threads_started":1}
I20260620 22:54:12.517247 14960 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45573
I20260620 22:54:12.517361 14960 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:12.517541 14960 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:12.517884 14610 ts_manager.cc:194] Registered new tserver with Master: 0b313404234d4884ab70d24297f36a7d (127.2.90.130:45819)
I20260620 22:54:12.518370 14610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:59571
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:54:12.541883 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45573
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:12.654551 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:12.654881 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:12.654965 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:12.656759 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:12.656952 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:12.657045 14965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:54:12.659140 14965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.90.148:56361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:12.659590 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:12.659900 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:12.666546 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.666529 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.666582 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:12.667698 14965 server_base.cc:1034] running on GCE node
I20260620 22:54:12.667906 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:12.668149 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:12.669310 14965 hybrid_clock.cc:648] HybridClock initialized: now 1781996052669288 us; error 34 us; skew 500 ppm
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:12.672124 14965 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:54:12.672577 14965 webserver.cc:460] Webserver started at http://127.2.90.131:40315/ using document root <none> and password file <none>
I20260620 22:54:12.672838 14965 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:12.672890 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:12.673012 14965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:12.687726 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "eeafd8a2c6f745f4bf0591ada567b31d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "770c2bcc69a8b397ebabe84bfa6d484c"
server_key_iv: "a8d4924dc7d790054128d2a4bd42067c"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.688174 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "eeafd8a2c6f745f4bf0591ada567b31d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "770c2bcc69a8b397ebabe84bfa6d484c"
server_key_iv: "a8d4924dc7d790054128d2a4bd42067c"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.691488 14965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 22:54:12.705904 14980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.706094 14965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:12.706171 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "eeafd8a2c6f745f4bf0591ada567b31d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "770c2bcc69a8b397ebabe84bfa6d484c"
server_key_iv: "a8d4924dc7d790054128d2a4bd42067c"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.706248 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:12.719889 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:12.739223 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:12.739457 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:12.739743 14965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:12.740163 14965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:12.740247 14965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.740317 14965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:12.740373 14965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.746476 14965 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:33389
I20260620 22:54:12.746973 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:54:12.755589 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 14965
I20260620 22:54:12.755671 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:54:12.755867 2410 external_mini_cluster.cc:1428] Setting key 5d2601e6438299bdc181c261d0476266
I20260620 22:54:12.756333 15093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:33389 every 8 connection(s)
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:12.767787 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:12.757489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:33313 (local address 127.2.90.190:45573)
0620 22:54:12.757537 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:12.757539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:12.757674 (+ 135us) server_negotiation.cc:407] Connection header received
0620 22:54:12.758351 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:12.758353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:12.758365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:12.758396 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:12.759200 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.759947 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.761513 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.761811 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.761852 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:12.764423 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:12.764424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:12.764425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:12.764432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:12.766055 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:12.766529 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:12.766530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:12.766531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:12.766586 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:12.766863 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:12.766864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:12.766864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:12.767011 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:12.767053 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:12.767616 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 22:54:12.767695 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:54:12.768493 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:12.757208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45573 (local address 127.2.90.131:33313)
0620 22:54:12.757547 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:12.757594 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:12.758223 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:12.758582 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:12.758586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:12.758822 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:12.759098 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:12.759104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.760111 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.760112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:12.761415 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:12.761418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:12.761952 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:12.761952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:12.762093 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:12.762696 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:12.762705 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:12.764311 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:12.766147 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:12.766153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:12.766155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:12.766443 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:12.766657 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:12.766657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:12.766658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:12.766785 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:12.767186 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:12.767192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:12.767366 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 22:54:12.768100 (+ 734us) client_negotiation.cc:240] Negotiation successful
0620 22:54:12.768257 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":80,"threads_started":1}
I20260620 22:54:12.769091 15094 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45573
I20260620 22:54:12.769178 15094 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:12.769351 15094 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:12.769760 14610 ts_manager.cc:194] Registered new tserver with Master: eeafd8a2c6f745f4bf0591ada567b31d (127.2.90.131:33389)
I20260620 22:54:12.770251 14610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:33313
I20260620 22:54:12.778477 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:54:12.779075 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 14691
I20260620 22:54:12.784835 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 14828
I20260620 22:54:12.790154 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 14965
I20260620 22:54:12.795326 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 14578
I20260620 22:54:12.800791 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45573
--webserver_interface=127.2.90.190
--webserver_port=35823
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:45573
--ranger_config_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:12.913216 15097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:12.913494 15097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:12.913522 15097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:12.915114 15097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:12.915167 15097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:12.915185 15097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:12.915208 15097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:12.915225 15097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:12.917191 15097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:45573
--ranger_config_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=35823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:12.917402 15097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:12.917598 15097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:12.921844 15105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.921901 15103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:12.921922 15102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:12.922149 15097 server_base.cc:1034] running on GCE node
I20260620 22:54:12.922375 15097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:12.922649 15097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:12.923794 15097 hybrid_clock.cc:648] HybridClock initialized: now 1781996052923784 us; error 25 us; skew 500 ppm
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:12.926055 15097 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:12.926465 15097 webserver.cc:460] Webserver started at http://127.2.90.190:35823/ using document root <none> and password file <none>
I20260620 22:54:12.926664 15097 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:12.926712 15097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:54:12 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996052, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:12.965307 15097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:12.966216 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:12.966391 15097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:12.966499 15097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "3b605533cde044eaa8702b2535620999"
format_stamp: "Formatted at 2026-06-20 22:54:11 on dist-test-slave-4mtw"
server_key: "48bafec7df1fff354615af57fa1a43ad"
server_key_iv: "35a2b90abc374431722218904de30d07"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:12.966950 15097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:12.979410 15097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:12.981109 15097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:12.981213 15097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:12.984969 15097 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:45573
I20260620 22:54:12.985035 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:45573 every 8 connection(s)
I20260620 22:54:12.985358 15097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:12.986380 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15097
I20260620 22:54:12.986649 15166 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:54:12.986646 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:46755
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:45573
--webserver_port=33359
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:12.987473 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Bootstrap starting.
I20260620 22:54:12.995131 15166 log.cc:824] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:12.996819 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:12.997059 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Bootstrap complete.
I20260620 22:54:12.999173 15166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:12.999392 15166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b605533cde044eaa8702b2535620999, State: Initialized, Role: FOLLOWER
I20260620 22:54:12.999555 15166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:12.999615 15166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:12.999653 15166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:12.999696 15166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:54:13.000669 15166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:13.000787 15166 leader_election.cc:302] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b605533cde044eaa8702b2535620999; no voters:
I20260620 22:54:13.000942 15166 leader_election.cc:288] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:54:13.000988 15170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:54:13.001147 15170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [term 2 LEADER]: Becoming Leader. State: Replica: 3b605533cde044eaa8702b2535620999, State: Running, Role: LEADER
I20260620 22:54:13.001195 15166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:13.001240 15170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } }
I20260620 22:54:13.001520 15171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b605533cde044eaa8702b2535620999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } } }
I20260620 22:54:13.001550 15170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b605533cde044eaa8702b2535620999. Latest consensus state: current_term: 2 leader_uuid: "3b605533cde044eaa8702b2535620999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b605533cde044eaa8702b2535620999" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45573 } } }
I20260620 22:54:13.001682 15170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:13.001667 15171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:13.002177 15177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:13.002651 15177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:13.002815 15177 catalog_manager.cc:1259] Loaded cluster ID: c038911e03ca490e9df1929c0872ebe9
I20260620 22:54:13.002882 15177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:54:13.002988 15166 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 22:54:13.004034 15177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:13.004208 15177 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Loaded TSK: 0
I20260620 22:54:13.004419 15177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:54:13.087607 15168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:13.087864 15168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:13.087924 15168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:13.089187 15168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:13.089227 15168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:13.089263 15168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:13.090881 15168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:46755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=33359
--tserver_master_addrs=127.2.90.190:45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:13.091110 15168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:13.091329 15168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:13.096169 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:13.096288 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:13.096798 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:13.097219 15168 server_base.cc:1034] running on GCE node
I20260620 22:54:13.097409 15168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:13.097640 15168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:13.098788 15168 hybrid_clock.cc:648] HybridClock initialized: now 1781996053098776 us; error 25 us; skew 500 ppm
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:13.101140 15168 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:13.101624 15168 webserver.cc:460] Webserver started at http://127.2.90.129:33359/ using document root <none> and password file <none>
I20260620 22:54:13.101848 15168 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:13.101908 15168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:13.121073 15168 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:13.124826 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.125047 15168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 22:54:13.125139 15168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "6593658f77bd4d5b86fcd0996930bcc2"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "b4b8ea80ae249ff919750b01c62dec2f"
server_key_iv: "40e98000f4cf45e837f3e8a0473320cf"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:13.126210 15168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:13.160003 15168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:13.162370 15168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:13.162547 15168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:13.162786 15168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:13.163148 15168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:13.163187 15168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.163216 15168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:13.163232 15168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.169903 15168 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:46755
I20260620 22:54:13.170400 15168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:54:13.171108 15327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:46755 every 8 connection(s)
I20260620 22:54:13.178309 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15168
I20260620 22:54:13.178550 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:45819
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:45573
--webserver_port=44793
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:13.182936 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.171131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45573 (local address 127.2.90.129:55423)
0620 22:54:13.172881 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:13.172931 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:13.173580 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:13.174018 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:13.174022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:13.174208 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:13.174454 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:13.174460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.175628 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.175629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:13.176397 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:13.176399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.177036 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.177037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:13.177156 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.177697 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:13.177706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:13.179160 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:13.181469 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:13.181474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:13.181477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:13.181676 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:13.181910 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:13.181911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:13.181911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:13.182014 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:13.182353 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:13.182358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:13.182489 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 22:54:13.182574 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:54:13.182719 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":110,"threads_started":1}
I20260620 22:54:13.182938 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.172852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:55423 (local address 127.2.90.190:45573)
0620 22:54:13.173082 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:13.173091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:13.173129 (+ 38us) server_negotiation.cc:407] Connection header received
0620 22:54:13.173796 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:13.173801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:13.173847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:13.173965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:13.174582 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.175532 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.176486 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.176941 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.177038 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.179278 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:13.179283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:13.179283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:13.179295 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:13.181385 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.181755 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.181759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.181759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.181853 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.182069 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.182070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.182070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.182292 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:13.182362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:13.182636 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 22:54:13.182739 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260620 22:54:13.184006 15328 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45573
I20260620 22:54:13.184101 15328 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.184274 15328 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
W20260620 22:54:13.185096 15328 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:45573 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:54:13.185158 15328 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.185217 15328 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:13.185487 15328 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.185561 15328 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
W20260620 22:54:13.185818 15328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:54:13.295827 15332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:13.296113 15332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:13.296154 15332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:13.297849 15332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:13.297909 15332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:13.297945 15332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:54:13.300027 15332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:45819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=44793
--tserver_master_addrs=127.2.90.190:45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:13.300282 15332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:13.300534 15332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:13.306583 15340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:13.306593 15337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:13.306711 15338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:13.306924 15332 server_base.cc:1034] running on GCE node
I20260620 22:54:13.307087 15332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:13.307298 15332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:13.308444 15332 hybrid_clock.cc:648] HybridClock initialized: now 1781996053308429 us; error 31 us; skew 500 ppm
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:13.311453 15332 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:54:13.311951 15332 webserver.cc:460] Webserver started at http://127.2.90.130:44793/ using document root <none> and password file <none>
I20260620 22:54:13.312191 15332 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:13.312282 15332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:13.333122 15332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:13.334069 15348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.334223 15332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:13.334301 15332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "0b313404234d4884ab70d24297f36a7d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "d87e9b3d704cd7a41c52db3fbabaa523"
server_key_iv: "4f805d56414d78808ec132ae89119771"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:13.334759 15332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:13.367868 15332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:13.369998 15332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:13.370160 15332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:13.370395 15332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:13.370762 15332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:13.370810 15332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.370841 15332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:13.370864 15332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.377897 15332 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45819
I20260620 22:54:13.378357 15332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:54:13.378996 15461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45819 every 8 connection(s)
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:13.387768 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.379467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:49673 (local address 127.2.90.190:45573)
0620 22:54:13.379541 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:13.379543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:13.379589 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:54:13.380089 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:13.380090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:13.380104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:13.380162 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:13.380779 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.381603 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.382583 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.382878 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.382970 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.384958 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:13.384959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:13.384960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:13.384967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:13.386361 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.386807 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.386808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.386809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.386870 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.387146 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.387147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.387148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.387283 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:13.387355 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:13.387594 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 22:54:13.387674 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 22:54:13.387833 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15332
I20260620 22:54:13.388082 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.90.148:56361/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:33389
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:45573
--webserver_port=40315
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:13.388139 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.379147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45573 (local address 127.2.90.130:49673)
0620 22:54:13.379465 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:13.379512 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:13.379992 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:13.380236 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:13.380240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:13.380408 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:13.380639 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:13.380645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.381712 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.381713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:13.382498 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:13.382500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.382985 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.382986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:13.383076 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.383572 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:13.383586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:13.384874 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:13.386434 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:13.386440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:13.386442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:13.386723 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:13.386939 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:13.386940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:13.386941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:13.387077 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:13.387349 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:13.387353 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:13.387465 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 22:54:13.387551 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:54:13.387683 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":83,"threads_started":1}
I20260620 22:54:13.388593 15462 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45573
I20260620 22:54:13.388696 15462 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.388861 15462 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
W20260620 22:54:13.389477 15462 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:45573 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:54:13.389552 15462 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.389617 15462 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:13.389990 15462 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.390065 15462 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
W20260620 22:54:13.390818 15462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:54:13.506348 15465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:13.506618 15465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:13.506657 15465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:13.508332 15465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:13.508394 15465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:13.508433 15465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:54:13.510607 15465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.90.148:56361/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:33389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=40315
--tserver_master_addrs=127.2.90.190:45573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:13.510851 15465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:13.511111 15465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:13.516088 15474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:13.516069 15472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:13.516258 15465 server_base.cc:1034] running on GCE node
W20260620 22:54:13.516388 15471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:13.516572 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:13.516837 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:13.517976 15465 hybrid_clock.cc:648] HybridClock initialized: now 1781996053517960 us; error 32 us; skew 500 ppm
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:13.520627 15465 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:54:13.521054 15465 webserver.cc:460] Webserver started at http://127.2.90.131:40315/ using document root <none> and password file <none>
I20260620 22:54:13.521277 15465 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:13.521337 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:13.545738 15465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:13.546615 15482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.546792 15465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:13.546877 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "eeafd8a2c6f745f4bf0591ada567b31d"
format_stamp: "Formatted at 2026-06-20 22:54:12 on dist-test-slave-4mtw"
server_key: "770c2bcc69a8b397ebabe84bfa6d484c"
server_key_iv: "a8d4924dc7d790054128d2a4bd42067c"
server_key_version: "kuduclusterkey@0"
I20260620 22:54:13.547214 15465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:13.562052 15465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:13.563838 15465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:13.563971 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:13.564188 15465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:13.564503 15465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:13.564539 15465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.564567 15465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:13.564590 15465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:13.570748 15465 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:33389
I20260620 22:54:13.570919 15595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:33389 every 8 connection(s)
I20260620 22:54:13.571184 15465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:54:13.576586 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15465
Jun 20 22:54:13 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996053, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:13.581950 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.571917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:60701 (local address 127.2.90.190:45573)
0620 22:54:13.571959 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:13.571961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:13.572188 (+ 227us) server_negotiation.cc:407] Connection header received
0620 22:54:13.572700 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:13.572701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:13.572714 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:13.572762 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:13.573406 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.574161 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.575522 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.575723 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.575787 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.578666 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:13.578668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:13.578668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:13.578675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:13.580157 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.580657 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.580658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.580658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.580738 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.581000 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.581001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.581002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.581130 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:13.581177 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:13.581780 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 22:54:13.581854 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:54:13.582659 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.571936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:45573 (local address 127.2.90.131:60701)
0620 22:54:13.572078 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:13.572112 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:13.572589 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:13.572846 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:13.572850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:13.573015 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:13.573288 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:13.573295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.574297 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.574298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:13.575442 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:13.575444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.575807 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.575809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:13.575939 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.576473 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:13.576483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:13.578562 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:13.580248 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:13.580254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:13.580256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:13.580568 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:13.580806 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:13.580806 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:13.580807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:13.580930 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:13.581221 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:13.581227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:13.581622 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 22:54:13.582268 (+ 646us) client_negotiation.cc:240] Negotiation successful
0620 22:54:13.582431 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260620 22:54:13.583248 15596 heartbeater.cc:347] Connected to a master server at 127.2.90.190:45573
I20260620 22:54:13.583353 15596 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.583526 15596 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
W20260620 22:54:13.584241 15596 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:45573 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 22:54:13.584331 15596 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.584394 15596 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:13.584712 15596 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:13.584789 15596 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
W20260620 22:54:13.585079 15596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:54:13.585572 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:13.577093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.2.90.190:45573)
0620 22:54:13.577358 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:13.577360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:13.577366 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:54:13.577385 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:13.577385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:13.577396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:13.577457 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:13.578212 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.579924 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.581028 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:13.581331 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:13.581393 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:13.582762 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:13.582764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:13.582764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:13.582783 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:13.584339 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.584719 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.584720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.584720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.584785 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:13.584985 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:13.584985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:13.584986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:13.585105 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:13.585253 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:13.585444 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 22:54:13.585507 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":49,"threads_started":1}
Jun 20 22:54:14 dist-test-slave-4mtw krb5kdc[14134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:14.185984 15328 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:14.186131 15328 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:14.186481 15130 ts_manager.cc:194] Registered new tserver with Master: 6593658f77bd4d5b86fcd0996930bcc2 (127.2.90.129:46755)
I20260620 22:54:14.187466 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:55423
I20260620 22:54:14.390975 15462 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:14.391119 15462 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:14.391369 15130 ts_manager.cc:194] Registered new tserver with Master: 0b313404234d4884ab70d24297f36a7d (127.2.90.130:45819)
I20260620 22:54:14.391885 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:49673
Jun 20 22:54:14 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:14.585253 15596 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:14.585402 15596 heartbeater.cc:511] Master 127.2.90.190:45573 requested a full tablet report, sending...
I20260620 22:54:14.585747 15130 ts_manager.cc:194] Registered new tserver with Master: eeafd8a2c6f745f4bf0591ada567b31d (127.2.90.131:33389)
I20260620 22:54:14.586268 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:60701
I20260620 22:54:14.593940 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:54:14.603080 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.594934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.2.90.190:45573)
0620 22:54:14.595236 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.595239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.595383 (+ 144us) server_negotiation.cc:407] Connection header received
0620 22:54:14.595405 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.595406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.595418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.595467 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:14.596155 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.596902 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.597924 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.598217 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.598274 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.599729 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:14.599731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:14.599731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:14.599738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:14.601521 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:14.601971 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:14.601973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:14.601973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:14.602096 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:14.602276 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:14.602276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:14.602277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:14.602401 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:14.602495 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.602901 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.602979 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":61,"threads_started":1}
I20260620 22:54:14.605334 15130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:54:14.607591 15130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:54:14.620260 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.610771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46755 (local address 127.0.0.1:59224)
0620 22:54:14.612107 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:14.612146 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:14.612202 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:14.612760 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:14.612764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:14.612766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:14.612842 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.612844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.614947 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.614948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.618976 (+ 4028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.618978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.619974 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.619975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.620055 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.620067 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:54:14.620128 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:54:14.620176 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":62,"threads_started":1}
I20260620 22:54:14.623389 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.610966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45819 (local address 127.0.0.1:50952)
0620 22:54:14.612316 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:14.612323 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:14.612352 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:14.614189 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:14.614189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:14.614191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:14.614241 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.614243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.617310 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.617311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.618394 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.618397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.623156 (+ 4759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.623157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.623229 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.623233 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:54:14.623276 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:54:14.623312 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":67,"threads_started":1}
I20260620 22:54:14.623880 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.611416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.2.90.129:46755)
0620 22:54:14.612462 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.612470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.612500 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:54:14.612545 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.612548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.612563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.612635 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:14.613060 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.614820 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.619047 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.619817 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.619884 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.619941 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.623757 (+ 3816us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.623805 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":63,"threads_started":1}
I20260620 22:54:14.624707 15262 tablet_service.cc:1467] Processing CreateTablet for tablet 6cca7d7dd6c44ca197131aee0a51fe00 (DEFAULT_TABLE table=test-table [id=d14bbe72d519451b8126a1a1e99e3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:14.625098 15262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cca7d7dd6c44ca197131aee0a51fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:14.625782 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.611596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.2.90.130:45819)
0620 22:54:14.613976 (+ 2380us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.613982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.613991 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:14.614024 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.614026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.614042 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.614110 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:14.616200 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.617219 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.622262 (+ 5043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.623002 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.623073 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.623124 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.625661 (+ 2537us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.625709 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2344,"thread_start_us":49,"threads_started":1}
I20260620 22:54:14.626437 15377 tablet_service.cc:1467] Processing CreateTablet for tablet 6cca7d7dd6c44ca197131aee0a51fe00 (DEFAULT_TABLE table=test-table [id=d14bbe72d519451b8126a1a1e99e3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:14.626746 15377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cca7d7dd6c44ca197131aee0a51fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:14.627429 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.611104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:33389 (local address 127.0.0.1:45640)
0620 22:54:14.612949 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:14.612959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:14.612994 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:14.620887 (+ 7893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:14.620888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:14.620890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:14.620926 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.620928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.624514 (+ 3586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.624515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.625556 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.625559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.627213 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.627214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.627272 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.627276 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:54:14.627323 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:54:14.627359 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1829,"thread_start_us":48,"threads_started":1}
I20260620 22:54:14.627712 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.611724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.2.90.131:33389)
0620 22:54:14.620648 (+ 8924us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.620653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.620664 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:54:14.620718 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.620720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.620735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.620807 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:14.621104 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.624413 (+ 3309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.626033 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.627054 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.627128 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.627179 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.627593 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.627635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8889,"thread_start_us":45,"threads_started":1}
I20260620 22:54:14.628306 15512 tablet_service.cc:1467] Processing CreateTablet for tablet 6cca7d7dd6c44ca197131aee0a51fe00 (DEFAULT_TABLE table=test-table [id=d14bbe72d519451b8126a1a1e99e3b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:14.628610 15512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cca7d7dd6c44ca197131aee0a51fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:14.631412 15631 tablet_bootstrap.cc:492] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Bootstrap starting.
I20260620 22:54:14.632055 15631 tablet_bootstrap.cc:654] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:14.632522 15631 log.cc:824] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:14.632926 15632 tablet_bootstrap.cc:492] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Bootstrap starting.
I20260620 22:54:14.633579 15632 tablet_bootstrap.cc:654] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:14.633937 15632 log.cc:824] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:14.634682 15631 tablet_bootstrap.cc:492] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: No bootstrap required, opened a new log
I20260620 22:54:14.634784 15631 ts_tablet_manager.cc:1397] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 22:54:14.637456 15631 raft_consensus.cc:348] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.637770 15631 raft_consensus.cc:374] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:14.637857 15631 raft_consensus.cc:729] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b313404234d4884ab70d24297f36a7d, State: Initialized, Role: FOLLOWER
I20260620 22:54:14.638088 15631 consensus_queue.cc:260] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.638437 15631 ts_tablet_manager.cc:1428] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260620 22:54:14.638592 15462 heartbeater.cc:503] Master 127.2.90.190:45573 was elected leader, sending a full tablet report...
I20260620 22:54:14.640054 15632 tablet_bootstrap.cc:492] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: No bootstrap required, opened a new log
I20260620 22:54:14.640193 15632 ts_tablet_manager.cc:1397] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 22:54:14.642894 15632 raft_consensus.cc:348] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.643126 15632 raft_consensus.cc:374] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:14.643198 15632 raft_consensus.cc:729] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeafd8a2c6f745f4bf0591ada567b31d, State: Initialized, Role: FOLLOWER
I20260620 22:54:14.643396 15632 consensus_queue.cc:260] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.643708 15632 ts_tablet_manager.cc:1428] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:54:14.643848 15596 heartbeater.cc:503] Master 127.2.90.190:45573 was elected leader, sending a full tablet report...
I20260620 22:54:14.644865 15630 tablet_bootstrap.cc:492] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: Bootstrap starting.
I20260620 22:54:14.645635 15630 tablet_bootstrap.cc:654] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:14.645983 15630 log.cc:824] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:14.649329 15630 tablet_bootstrap.cc:492] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: No bootstrap required, opened a new log
I20260620 22:54:14.649399 15630 ts_tablet_manager.cc:1397] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 22:54:14.652001 15630 raft_consensus.cc:348] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.652248 15630 raft_consensus.cc:374] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:14.652324 15630 raft_consensus.cc:729] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6593658f77bd4d5b86fcd0996930bcc2, State: Initialized, Role: FOLLOWER
I20260620 22:54:14.652505 15630 consensus_queue.cc:260] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.652858 15630 ts_tablet_manager.cc:1428] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:54:14.653008 15328 heartbeater.cc:503] Master 127.2.90.190:45573 was elected leader, sending a full tablet report...
W20260620 22:54:14.671649 15329 tablet.cc:2369] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:54:14.690410 15638 raft_consensus.cc:482] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:54:14.690567 15638 raft_consensus.cc:504] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.690955 15638 leader_election.cc:288] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b313404234d4884ab70d24297f36a7d (127.2.90.130:45819), eeafd8a2c6f745f4bf0591ada567b31d (127.2.90.131:33389)
I20260620 22:54:14.696283 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.691552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:55829 (local address 127.2.90.131:33389)
0620 22:54:14.691596 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.691598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.691608 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:54:14.691706 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.691708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.691720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.691747 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:14.691959 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.692710 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.695258 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.695800 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.695875 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.695884 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.696172 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.696215 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:54:14.696789 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.691374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:33389 (local address 127.2.90.129:55829)
0620 22:54:14.691565 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:14.691574 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:14.691624 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:14.691809 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:14.691810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:14.691812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:14.691885 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.691887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.692805 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.692805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.695164 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.695167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.695907 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.695908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.695970 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.695973 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:54:14.696638 (+ 665us) client_negotiation.cc:240] Negotiation successful
0620 22:54:14.696698 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1267,"thread_start_us":64,"threads_started":1}
I20260620 22:54:14.697046 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.691242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:46149 (local address 127.2.90.130:45819)
0620 22:54:14.691289 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.691292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.691432 (+ 140us) server_negotiation.cc:407] Connection header received
0620 22:54:14.691639 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.691640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.691654 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.691685 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:14.691940 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.692658 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.695273 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.695786 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.695843 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.695852 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.696947 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.696991 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:54:14.697641 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cca7d7dd6c44ca197131aee0a51fe00" candidate_uuid: "6593658f77bd4d5b86fcd0996930bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" is_pre_election: true
I20260620 22:54:14.697788 15550 raft_consensus.cc:2454] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6593658f77bd4d5b86fcd0996930bcc2 in term 0.
I20260620 22:54:14.698015 15218 leader_election.cc:302] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6593658f77bd4d5b86fcd0996930bcc2, eeafd8a2c6f745f4bf0591ada567b31d; no voters:
I20260620 22:54:14.698143 15638 raft_consensus.cc:2781] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:54:14.698251 15638 raft_consensus.cc:482] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:54:14.698278 15638 raft_consensus.cc:3037] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:14.697364 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.691146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45819 (local address 127.2.90.129:46149)
0620 22:54:14.691358 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:14.691368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:14.691460 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:14.691757 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:14.691758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:14.691761 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:14.691851 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.691854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.692785 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.692786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.694106 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:14.694109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.696051 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.696051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:14.696106 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.696108 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:54:14.697258 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0620 22:54:14.697297 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":111,"threads_started":1}
I20260620 22:54:14.698671 15409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cca7d7dd6c44ca197131aee0a51fe00" candidate_uuid: "6593658f77bd4d5b86fcd0996930bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b313404234d4884ab70d24297f36a7d" is_pre_election: true
I20260620 22:54:14.698808 15409 raft_consensus.cc:2454] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6593658f77bd4d5b86fcd0996930bcc2 in term 0.
I20260620 22:54:14.699165 15638 raft_consensus.cc:504] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.699476 15409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cca7d7dd6c44ca197131aee0a51fe00" candidate_uuid: "6593658f77bd4d5b86fcd0996930bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b313404234d4884ab70d24297f36a7d"
I20260620 22:54:14.699549 15409 raft_consensus.cc:3037] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:14.700404 15409 raft_consensus.cc:2454] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6593658f77bd4d5b86fcd0996930bcc2 in term 1.
I20260620 22:54:14.700623 15218 leader_election.cc:302] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b313404234d4884ab70d24297f36a7d, 6593658f77bd4d5b86fcd0996930bcc2; no voters:
I20260620 22:54:14.700721 15638 leader_election.cc:288] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [CANDIDATE]: Term 1 election: Requested vote from peers 0b313404234d4884ab70d24297f36a7d (127.2.90.130:45819), eeafd8a2c6f745f4bf0591ada567b31d (127.2.90.131:33389)
I20260620 22:54:14.700765 15638 raft_consensus.cc:2781] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:14.700793 15638 raft_consensus.cc:686] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [term 1 LEADER]: Becoming Leader. State: Replica: 6593658f77bd4d5b86fcd0996930bcc2, State: Running, Role: LEADER
I20260620 22:54:14.700866 15638 consensus_queue.cc:237] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } }
I20260620 22:54:14.701074 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cca7d7dd6c44ca197131aee0a51fe00" candidate_uuid: "6593658f77bd4d5b86fcd0996930bcc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeafd8a2c6f745f4bf0591ada567b31d"
I20260620 22:54:14.701169 15550 raft_consensus.cc:3037] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:14.701778 15130 catalog_manager.cc:5495] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6593658f77bd4d5b86fcd0996930bcc2 (127.2.90.129). New cstate: current_term: 1 leader_uuid: "6593658f77bd4d5b86fcd0996930bcc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6593658f77bd4d5b86fcd0996930bcc2" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 } health_report { overall_health: UNKNOWN } } }
I20260620 22:54:14.701998 15550 raft_consensus.cc:2454] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6593658f77bd4d5b86fcd0996930bcc2 in term 1.
Jun 20 22:54:14 dist-test-slave-4mtw krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for HTTP/127.2.90.148@KRBTEST.COM
I20260620 22:54:14.731454 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.728807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.2.90.129:46755)
0620 22:54:14.728880 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:14.728882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:14.728889 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:54:14.728929 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:14.728930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:14.728956 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:14.729021 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:54:14.729214 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.729948 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.730694 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:14.730934 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:14.730984 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:14.731142 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:54:14.731248 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:54:14.731318 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:14.731338 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 22:54:14.731374 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 22:54:14.734058 15409 raft_consensus.cc:1261] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 1 FOLLOWER]: Refusing update from remote peer 6593658f77bd4d5b86fcd0996930bcc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:14.734073 15550 raft_consensus.cc:1261] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 1 FOLLOWER]: Refusing update from remote peer 6593658f77bd4d5b86fcd0996930bcc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:14.734372 15638 consensus_queue.cc:1035] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b313404234d4884ab70d24297f36a7d" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:14.734652 15638 consensus_queue.cc:1035] T 6cca7d7dd6c44ca197131aee0a51fe00 P 6593658f77bd4d5b86fcd0996930bcc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeafd8a2c6f745f4bf0591ada567b31d" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:14.737663 15645 mvcc.cc:205] Tried to move back new op lower bound from 7299055840186974208 to 7299055840055336960. Current Snapshot: MvccSnapshot[applied={T|T < 7299055840186974208}]
I20260620 22:54:14.738513 15646 mvcc.cc:205] Tried to move back new op lower bound from 7299055840186974208 to 7299055840055336960. Current Snapshot: MvccSnapshot[applied={T|T < 7299055840186974208}]
I20260620 22:54:14.739321 15648 mvcc.cc:205] Tried to move back new op lower bound from 7299055840186974208 to 7299055840055336960. Current Snapshot: MvccSnapshot[applied={T|T < 7299055840186974208}]
I20260620 22:54:14.741389 15130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:14.741487 15130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:14.742205 15130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b605533cde044eaa8702b2535620999: Sending DeleteTablet for 3 replicas of tablet 6cca7d7dd6c44ca197131aee0a51fe00
I20260620 22:54:14.742693 15262 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cca7d7dd6c44ca197131aee0a51fe00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:14 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:59224
I20260620 22:54:14.742908 15377 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cca7d7dd6c44ca197131aee0a51fe00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:14 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:50952
I20260620 22:54:14.743117 15512 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cca7d7dd6c44ca197131aee0a51fe00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:14 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:45640
I20260620 22:54:14.743754 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15168
I20260620 22:54:14.744563 15655 tablet_replica.cc:330] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: stopping tablet replica
I20260620 22:54:14.744659 15655 raft_consensus.cc:2229] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:14.744774 15655 raft_consensus.cc:2258] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:14.744980 15656 tablet_replica.cc:330] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: stopping tablet replica
I20260620 22:54:14.745060 15656 raft_consensus.cc:2229] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:14.745149 15656 raft_consensus.cc:2258] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:14.745573 15656 ts_tablet_manager.cc:1905] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:14.747044 15656 ts_tablet_manager.cc:1918] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:14.747098 15656 log.cc:1196] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/6cca7d7dd6c44ca197131aee0a51fe00
I20260620 22:54:14.747314 15656 ts_tablet_manager.cc:1939] T 6cca7d7dd6c44ca197131aee0a51fe00 P eeafd8a2c6f745f4bf0591ada567b31d: Deleting consensus metadata
I20260620 22:54:14.748054 15117 catalog_manager.cc:4841] TS eeafd8a2c6f745f4bf0591ada567b31d (127.2.90.131:33389): tablet 6cca7d7dd6c44ca197131aee0a51fe00 (table test-table [id=d14bbe72d519451b8126a1a1e99e3b01]) successfully deleted
I20260620 22:54:14.749440 15655 ts_tablet_manager.cc:1905] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:14.751021 15655 ts_tablet_manager.cc:1918] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:14.751087 15655 log.cc:1196] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/6cca7d7dd6c44ca197131aee0a51fe00
I20260620 22:54:14.751315 15655 ts_tablet_manager.cc:1939] T 6cca7d7dd6c44ca197131aee0a51fe00 P 0b313404234d4884ab70d24297f36a7d: Deleting consensus metadata
I20260620 22:54:14.755232 15117 catalog_manager.cc:4841] TS 0b313404234d4884ab70d24297f36a7d (127.2.90.130:45819): tablet 6cca7d7dd6c44ca197131aee0a51fe00 (table test-table [id=d14bbe72d519451b8126a1a1e99e3b01]) successfully deleted
W20260620 22:54:14.755445 15116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:46755 (error 108)
I20260620 22:54:14.756075 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15332
I20260620 22:54:14.756086 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:14.755947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46755 (local address 127.0.0.1:59254)
0620 22:54:14.755998 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:14.756049 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:54:14.761297 15116 catalog_manager.cc:4584] TS 6593658f77bd4d5b86fcd0996930bcc2 (127.2.90.129:46755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cca7d7dd6c44ca197131aee0a51fe00: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46755: connect: Connection refused (error 111)
I20260620 22:54:14.763427 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15465
I20260620 22:54:14.768671 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15097
I20260620 22:54:14.774264 2410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 22:54:15.107430 2410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 22:54:15.107518 2410 mini_ranger.cc:66] Stopping Ranger...
I20260620 22:54:15.440621 2410 mini_ranger.cc:68] Stopped Ranger
2026-06-20 22:54:15.440 UTC [14164] LOG: received smart shutdown request
2026-06-20 22:54:15.443 UTC [14164] LOG: background worker "logical replication launcher" (PID 14173) exited with exit code 1
2026-06-20 22:54:15.443 UTC [14168] LOG: shutting down
2026-06-20 22:54:15.572 UTC [14164] LOG: database system is shut down
2026-06-20T22:54:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44834 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:54:15 dist-test-slave-4mtw krb5kdc[15667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:54:15 dist-test-slave-4mtw krb5kdc[15667](info): set up 2 sockets
Jun 20 22:54:15 dist-test-slave-4mtw krb5kdc[15667](info): commencing operation
krb5kdc: starting...
W20260620 22:54:16.604279 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:54:16 dist-test-slave-4mtw krb5kdc[15667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:54:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:54:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:54:16.709226 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42239
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:42239,127.2.90.189:38383,127.2.90.188:35349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:16.785866 15680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:16.786051 15680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:16.786072 15680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:16.787209 15680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:16.787249 15680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:16.787262 15680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:16.787292 15680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:16.787315 15680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:16.788795 15680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:42239,127.2.90.189:38383,127.2.90.188:35349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:16.788996 15680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:16.789165 15680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:16.793496 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:16.793509 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:16.793594 15680 server_base.cc:1034] running on GCE node
W20260620 22:54:16.793511 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:16.793893 15680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:16.794153 15680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:16.795298 15680 hybrid_clock.cc:648] HybridClock initialized: now 1781996056795288 us; error 29 us; skew 500 ppm
Jun 20 22:54:16 dist-test-slave-4mtw krb5kdc[15667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:16.801589 15680 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:16.802012 15680 webserver.cc:460] Webserver started at http://127.2.90.190:45193/ using document root <none> and password file <none>
I20260620 22:54:16.802217 15680 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:16.802263 15680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:16.802400 15680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:16.803402 15680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "47eda3e3daf540288980ac57958bd471"
format_stamp: "Formatted at 2026-06-20 22:54:16 on dist-test-slave-4mtw"
server_key: "6f1bc3bae2898f1a84f2fee599b036e8"
server_key_iv: "7d2ab83829acb2e9126866da0f5c3b15"
server_key_version: "encryptionkey@0"
I20260620 22:54:16.803712 15680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "47eda3e3daf540288980ac57958bd471"
format_stamp: "Formatted at 2026-06-20 22:54:16 on dist-test-slave-4mtw"
server_key: "6f1bc3bae2898f1a84f2fee599b036e8"
server_key_iv: "7d2ab83829acb2e9126866da0f5c3b15"
server_key_version: "encryptionkey@0"
I20260620 22:54:16.804896 15680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:54:16.805496 15695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:16.805662 15680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:16.805744 15680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "47eda3e3daf540288980ac57958bd471"
format_stamp: "Formatted at 2026-06-20 22:54:16 on dist-test-slave-4mtw"
server_key: "6f1bc3bae2898f1a84f2fee599b036e8"
server_key_iv: "7d2ab83829acb2e9126866da0f5c3b15"
server_key_version: "encryptionkey@0"
I20260620 22:54:16.805800 15680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:16.827334 15680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:16.828997 15680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:16.829124 15680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:16.832748 15680 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:42239
I20260620 22:54:16.832777 15747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:42239 every 8 connection(s)
I20260620 22:54:16.833094 15680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:16.833876 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:16.835312 15748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:54:16 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:16.842948 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15680
I20260620 22:54:16.843055 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:54:16.843221 2410 external_mini_cluster.cc:1428] Setting key 4531e990c8a3a530aed8d4cfb39a1cc2
I20260620 22:54:16.843446 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.835909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42239 (local address 127.0.0.1:48856)
0620 22:54:16.836058 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.836094 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:16.836539 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:16.836772 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:16.836774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:16.837003 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:16.837149 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:16.837153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.838075 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.838076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:16.838762 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:16.838763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.839188 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.839188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:16.839237 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:16.839624 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:16.839630 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:16.840904 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:16.842199 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:16.842205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:16.842206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:16.842381 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:16.842552 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:16.842553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:16.842553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:16.842619 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:16.842933 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:16.842936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:16.842991 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 22:54:16.843061 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:54:16.843214 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260620 22:54:16.843447 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.835932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.2.90.190:42239)
0620 22:54:16.836093 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:16.836102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:16.836174 (+ 72us) server_negotiation.cc:407] Connection header received
0620 22:54:16.836647 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:16.836649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:16.836667 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:16.836753 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:16.837215 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.837995 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.838824 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.839134 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.839206 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:16.840984 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:16.840990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:16.840991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:16.840996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:16.842137 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:16.842439 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:16.842441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:16.842441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:16.842508 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:16.842660 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:16.842660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:16.842660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:16.842872 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:16.842946 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:16.843141 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 22:54:16.843284 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260620 22:54:16.844087 15748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:54:16.844403 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.844274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35882)
0620 22:54:16.844321 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.844375 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 22:54:16.844537 15696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
I20260620 22:54:16.844914 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.844827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35888)
0620 22:54:16.844866 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.844891 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:54:16.845016 15748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:38383: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.90.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab.
Entry for principal kudu/127.2.90.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab.
I20260620 22:54:16.862880 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:38383
--webserver_interface=127.2.90.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab
--principal=kudu/127.2.90.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:42239,127.2.90.189:38383,127.2.90.188:35349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:16.894209 15748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } attempt: 1
I20260620 22:54:16.894639 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.894520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35904)
0620 22:54:16.894574 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.894606 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:54:16.895012 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.894925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35916)
0620 22:54:16.894967 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.894994 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 22:54:16.895115 15748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:38383: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
W20260620 22:54:16.940366 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:16.940557 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:16.940578 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:16.941669 15755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:16.941708 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:16.941721 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:16.941732 15755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:16.941745 15755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:16.943121 15755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:42239,127.2.90.189:38383,127.2.90.188:35349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab
--principal=kudu/127.2.90.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.90.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:16.943308 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:16.943475 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:16.947943 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:16.947997 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:16.948119 15755 server_base.cc:1034] running on GCE node
W20260620 22:54:16.948007 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:16.948418 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:16.948712 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:16.949854 15755 hybrid_clock.cc:648] HybridClock initialized: now 1781996056949845 us; error 25 us; skew 500 ppm
Jun 20 22:54:16 dist-test-slave-4mtw krb5kdc[15667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:16.952047 15755 init.cc:377] Logged in from keytab as kudu/127.2.90.189@KRBTEST.COM (short username kudu)
I20260620 22:54:16.952486 15755 webserver.cc:460] Webserver started at http://127.2.90.189:42029/ using document root <none> and password file <none>
I20260620 22:54:16.952672 15755 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:16.952760 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:16.952881 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:16.953855 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data/instance:
uuid: "11d0f122dc654073882331dd391d1c44"
format_stamp: "Formatted at 2026-06-20 22:54:16 on dist-test-slave-4mtw"
server_key: "68f425bffb321dc83d290edadec4d9ee"
server_key_iv: "7c07f691a2195d2e4605f087bdb6c2cb"
server_key_version: "encryptionkey@0"
I20260620 22:54:16.954169 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal/instance:
uuid: "11d0f122dc654073882331dd391d1c44"
format_stamp: "Formatted at 2026-06-20 22:54:16 on dist-test-slave-4mtw"
server_key: "68f425bffb321dc83d290edadec4d9ee"
server_key_iv: "7c07f691a2195d2e4605f087bdb6c2cb"
server_key_version: "encryptionkey@0"
I20260620 22:54:16.955322 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:54:16.955926 15770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:16.956059 15755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:16.956148 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal
uuid: "11d0f122dc654073882331dd391d1c44"
format_stamp: "Formatted at 2026-06-20 22:54:16 on dist-test-slave-4mtw"
server_key: "68f425bffb321dc83d290edadec4d9ee"
server_key_iv: "7c07f691a2195d2e4605f087bdb6c2cb"
server_key_version: "encryptionkey@0"
I20260620 22:54:16.956215 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:16.963277 15748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } attempt: 2
I20260620 22:54:16.963649 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.963539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35930)
0620 22:54:16.963586 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.963621 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:54:16.963897 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.963834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35940)
0620 22:54:16.963863 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.963884 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 22:54:16.963984 15748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:38383: Network error: Client connection negotiation failed: client connection to 127.2.90.189:38383: connect: Connection refused (error 111)
I20260620 22:54:16.978856 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:16.980526 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:16.980650 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:16.984253 15755 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.189:38383
I20260620 22:54:16.984289 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.189:38383 every 8 connection(s)
I20260620 22:54:16.984598 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
I20260620 22:54:16.985426 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:16.986569 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15755
I20260620 22:54:16.986676 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-1/wal/instance
I20260620 22:54:16.986896 2410 external_mini_cluster.cc:1428] Setting key 42de0f95d11837e2170324f0f4eef3c4
I20260620 22:54:16.987162 15823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:54:16 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.188@KRBTEST.COM" created.
I20260620 22:54:16.995851 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.987928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.2.90.190:42239)
0620 22:54:16.987988 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:16.987990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:16.988215 (+ 225us) server_negotiation.cc:407] Connection header received
0620 22:54:16.988701 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:16.988702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:16.988714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:16.988772 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:16.989423 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.989896 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.990947 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.991156 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.991213 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:16.993209 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:16.993210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:16.993211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:16.993219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:16.994518 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:16.994901 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:16.994902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:16.994902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:16.994989 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:16.995206 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:16.995207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:16.995207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:16.995315 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:16.995407 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:16.995678 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 22:54:16.995761 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:54:16.996012 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.987872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42239 (local address 127.0.0.1:48870)
0620 22:54:16.988098 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.988133 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:16.988588 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:16.988841 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:16.988844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:16.989085 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:16.989339 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:16.989345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.989977 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.989979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:16.990845 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:16.990848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.991226 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.991227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:16.991356 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:16.991809 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:16.991824 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:16.993123 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:16.994592 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:16.994599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:16.994602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:16.994811 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:16.995045 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:16.995046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:16.995046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:16.995151 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:16.995393 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:16.995399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:16.995548 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 22:54:16.995647 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 22:54:16.995796 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260620 22:54:16.996536 15823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:54:16 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:54:17.002905 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.996797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35942)
0620 22:54:16.996842 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:16.996851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:16.996885 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:16.997141 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:16.997143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:16.997280 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:16.997338 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:16.997339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.998105 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.998106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:16.998620 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:16.998622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.999038 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.999039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:16.999097 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:16.999429 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:16.999434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.000501 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.001986 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.001987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.001987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.002133 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.002300 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.002301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.002301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.002353 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.002588 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.002589 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.002624 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.002684 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.002777 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:54:17.002962 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:16.996874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.2.90.189:38383)
0620 22:54:16.997007 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:16.997011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:16.997030 (+ 19us) server_negotiation.cc:407] Connection header received
0620 22:54:16.997050 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:16.997052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:16.997064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:16.997119 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:16.997407 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.998035 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.998690 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:16.998978 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:16.999035 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.000570 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.000578 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.000578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.000584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.001914 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.002197 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.002200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.002200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.002251 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.002402 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.002403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.002403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.002530 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.002598 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.002711 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.002826 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260620 22:54:17.003602 15823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:54:17.003914 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.003814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42456)
0620 22:54:17.003852 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.003895 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 22:54:17.004029 15771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
I20260620 22:54:17.004390 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.004315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42472)
0620 22:54:17.004345 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.004369 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 22:54:17.004478 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:35349: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab.
Entry for principal kudu/127.2.90.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab.
I20260620 22:54:17.009696 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:35349
--webserver_interface=127.2.90.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab
--principal=kudu/127.2.90.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:37199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:42239,127.2.90.189:38383,127.2.90.188:35349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:17.053689 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } attempt: 1
I20260620 22:54:17.054149 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.054035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42484)
0620 22:54:17.054089 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.054120 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:54:17.054539 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.054444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42492)
0620 22:54:17.054495 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.054520 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 22:54:17.054638 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:35349: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
I20260620 22:54:17.055114 15748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } attempt: 3
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:54:17.061708 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.055401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.2.90.189:38383)
0620 22:54:17.055440 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.055441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.055452 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:54:17.055510 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.055511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.055521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.055570 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.055865 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.056356 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.057057 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.057262 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.057316 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.059347 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.059348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.059348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.059355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.060748 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.061097 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.061097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.061098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.061144 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.061303 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.061303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.061304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.061397 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.061438 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.061586 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.061644 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:54:17.061759 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.055350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35950)
0620 22:54:17.055399 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.055409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:17.055457 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:17.055579 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:17.055580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:17.055729 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:17.055798 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.055799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.056429 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.056430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.056986 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.056988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.057319 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.057321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.057388 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.057828 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:17.057832 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.059270 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.060817 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.060818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.060818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.061033 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.061193 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.061193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.061194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.061252 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.061445 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.061446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.061499 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.061561 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.061658 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:54:17.062155 15748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:54:17.062465 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.062360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42500)
0620 22:54:17.062405 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.062439 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:54:17.062733 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.062664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42508)
0620 22:54:17.062696 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.062717 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 22:54:17.062815 15748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:35349: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
W20260620 22:54:17.089470 15830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:17.089669 15830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:17.089691 15830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:17.090811 15830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:17.090859 15830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:17.090879 15830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:17.090906 15830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:17.090922 15830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:17.092368 15830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:37199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:42239,127.2.90.189:38383,127.2.90.188:35349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab
--principal=kudu/127.2.90.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:35349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.90.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:17.092555 15830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:17.092762 15830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:54:17.093953 15748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } attempt: 1
I20260620 22:54:17.094350 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.094237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42524)
0620 22:54:17.094290 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.094321 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:54:17.094646 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.094571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42534)
0620 22:54:17.094605 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.094629 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 22:54:17.094726 15748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:35349: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
W20260620 22:54:17.097218 15838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:17.097230 15836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:17.097278 15835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:17.097390 15830 server_base.cc:1034] running on GCE node
I20260620 22:54:17.097615 15830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:17.097867 15830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:17.098990 15830 hybrid_clock.cc:648] HybridClock initialized: now 1781996057098987 us; error 34 us; skew 500 ppm
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:17.101217 15830 init.cc:377] Logged in from keytab as kudu/127.2.90.188@KRBTEST.COM (short username kudu)
I20260620 22:54:17.101631 15830 webserver.cc:460] Webserver started at http://127.2.90.188:36015/ using document root <none> and password file <none>
I20260620 22:54:17.101823 15830 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:17.101881 15830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:17.102000 15830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:17.102995 15830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data/instance:
uuid: "40627acda2c8463c8b14137ef44acc59"
format_stamp: "Formatted at 2026-06-20 22:54:17 on dist-test-slave-4mtw"
server_key: "0a0c1e3490d3ea05273c92e537e01a49"
server_key_iv: "2d1eee1493dc4c664ccb3360ef116b94"
server_key_version: "encryptionkey@0"
I20260620 22:54:17.103308 15830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal/instance:
uuid: "40627acda2c8463c8b14137ef44acc59"
format_stamp: "Formatted at 2026-06-20 22:54:17 on dist-test-slave-4mtw"
server_key: "0a0c1e3490d3ea05273c92e537e01a49"
server_key_iv: "2d1eee1493dc4c664ccb3360ef116b94"
server_key_version: "encryptionkey@0"
I20260620 22:54:17.104501 15830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:54:17.105189 15845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:17.105350 15830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:17.105432 15830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal
uuid: "40627acda2c8463c8b14137ef44acc59"
format_stamp: "Formatted at 2026-06-20 22:54:17 on dist-test-slave-4mtw"
server_key: "0a0c1e3490d3ea05273c92e537e01a49"
server_key_iv: "2d1eee1493dc4c664ccb3360ef116b94"
server_key_version: "encryptionkey@0"
I20260620 22:54:17.105496 15830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:17.122797 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } attempt: 2
I20260620 22:54:17.123229 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.123089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42550)
0620 22:54:17.123148 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.123191 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:54:17.123588 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.123507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42560)
0620 22:54:17.123553 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.123573 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 22:54:17.123674 15823 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:35349: Network error: Client connection negotiation failed: client connection to 127.2.90.188:35349: connect: Connection refused (error 111)
I20260620 22:54:17.127457 15830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:17.129177 15830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:17.129300 15830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:17.133023 15830 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.188:35349
I20260620 22:54:17.133062 15897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.188:35349 every 8 connection(s)
I20260620 22:54:17.133389 15830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
I20260620 22:54:17.133826 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15830
I20260620 22:54:17.133950 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781995912840861-2410-0/minicluster-data/master-2/wal/instance
I20260620 22:54:17.134155 2410 external_mini_cluster.cc:1428] Setting key 2026341ebaf9c02f0d16b8cf1dca3063
I20260620 22:54:17.134371 15898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:17.136255 15898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:17.144004 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.134619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.2.90.188:35349)
0620 22:54:17.134798 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.134806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.134840 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:54:17.135345 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.135351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.135404 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.135521 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.136116 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.137207 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.138099 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.138600 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.138706 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.140849 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.140855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.140855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.140861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.142506 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.142854 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.142858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.142858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.142951 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.143112 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.143112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.143113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.143388 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.143507 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.143622 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.143824 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260620 22:54:17.144207 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.136861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.2.90.190:42239)
0620 22:54:17.136908 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.136910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.137008 (+ 98us) server_negotiation.cc:407] Connection header received
0620 22:54:17.137067 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.137067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.137075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.137118 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.137572 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.138220 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.139107 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.139288 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.139339 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.141407 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.141409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.141409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.141417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.143214 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.143521 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.143522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.143523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.143578 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.143727 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.143728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.143728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.143835 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.143889 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.144046 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.144119 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:54:17.144218 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.136777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42239 (local address 127.0.0.1:48882)
0620 22:54:17.136912 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.136951 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:17.137013 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:17.137143 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:17.137145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:17.137439 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:17.137496 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.137498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.138291 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.138292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.139020 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.139023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.139360 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.139361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.139428 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.139820 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:17.139828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.141328 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.143274 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.143280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.143281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.143447 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.143630 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.143631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.143631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.143678 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.143886 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.143893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.143937 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.144014 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.144115 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260620 22:54:17.144703 15898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:17.150367 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.144911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:38383 (local address 127.0.0.1:35952)
0620 22:54:17.144946 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.144956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:17.144987 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:17.145125 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:17.145125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:17.145253 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:17.145287 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.145288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.145854 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.145855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.146318 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.146319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.146642 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.146642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.146694 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.147007 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:17.147012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.148089 (+ 1077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.149539 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.149540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.149540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.149682 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.149862 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.149862 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.149863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.149900 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.150105 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.150106 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.150147 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.150225 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.150293 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 22:54:17.150411 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.144983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.2.90.189:38383)
0620 22:54:17.145022 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.145023 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.145031 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:54:17.145053 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.145054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.145062 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.145103 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.145346 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.145789 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.146386 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.146588 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.146639 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.148164 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.148165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.148166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.148171 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.149474 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.149746 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.149747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.149748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.149801 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.149934 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.149935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.149935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.150051 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.150096 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.150260 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.150337 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:54:17.150709 15898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:54:17.151117 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.144739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.2.90.190:42239)
0620 22:54:17.144827 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.144829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.144837 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:54:17.144854 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.144854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.144864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.144915 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.145454 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.145882 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.146479 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.146651 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.146699 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.148752 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.148754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.148754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.148760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.149946 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.150239 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.150239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.150240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.150290 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.150498 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.150499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.150500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.150620 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.150778 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.150950 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.151034 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:54:17.156600 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.150909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42564)
0620 22:54:17.151188 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.151197 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:17.151242 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:17.151410 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:17.151411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:17.151540 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:17.151568 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.151570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.152176 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.152176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.152638 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.152640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.152978 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.152979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.153022 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.153358 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:17.153362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.154372 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.155764 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.155765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.155765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.155908 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.156079 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.156080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.156081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.156126 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.156315 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.156316 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.156351 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.156408 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.156504 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260620 22:54:17.156651 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.150951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.2.90.188:35349)
0620 22:54:17.150981 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.150982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.151248 (+ 266us) server_negotiation.cc:407] Connection header received
0620 22:54:17.151287 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.151288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.151296 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.151326 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.151629 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.152113 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.152730 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.152927 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.152975 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.154446 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.154447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.154447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.154453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.155698 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.155979 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.155979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.155980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.156030 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.156173 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.156173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.156174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.156267 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.156319 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.156434 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.156552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":15}
I20260620 22:54:17.157919 15898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59: Bootstrap starting.
I20260620 22:54:17.158326 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.151773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.2.90.189:38383)
0620 22:54:17.151809 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.151811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.151816 (+ 5us) server_negotiation.cc:407] Connection header received
0620 22:54:17.151832 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.151833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.151839 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.151877 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.152295 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.152778 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.153494 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.153681 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.153731 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.155945 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.155946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.155947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.155952 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.157365 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.157636 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.157637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.157637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.157685 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.157843 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.157844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.157844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.157938 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.158114 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.158149 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.158213 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:54:17.158860 15898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:17.159139 15898 log.cc:824] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:17.159559 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:54:17.159968 15898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59: No bootstrap required, opened a new log
I20260620 22:54:17.161969 15898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.162122 15898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:17.162158 15898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40627acda2c8463c8b14137ef44acc59, State: Initialized, Role: FOLLOWER
I20260620 22:54:17.162276 15898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.162554 15898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:17.163913 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.165253 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:54:17.164302 15922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:54:17.165412 15922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:54:17.169853 15748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } attempt: 2
I20260620 22:54:17.172125 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.160982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.2.90.190:42239)
0620 22:54:17.164859 (+ 3877us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.164861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.164871 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:54:17.164895 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.164896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.164906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.164963 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.165899 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.166567 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.167575 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.167852 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.167910 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.169307 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.169309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.169310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.169317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.170785 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.171165 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.171166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.171166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.171221 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.171419 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.171420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.171420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.171541 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.171623 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.171959 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.172032 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3848}
I20260620 22:54:17.170739 15830 master_runner.cc:186] Error getting master registration for 127.2.90.190:42239: OK, instance_id { permanent_uuid: "47eda3e3daf540288980ac57958bd471" instance_seqno: 1781996056831695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:54:17.173142 15830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:54:17.173497 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.163790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.2.90.188:35349)
0620 22:54:17.165018 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.165020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.165027 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:54:17.165044 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.165045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.165053 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.165096 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.165899 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.166526 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.167540 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.167803 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.167863 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.169452 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.169453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.169454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.169460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.171470 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.171831 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.171832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.171832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.171882 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.172056 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.172057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.172057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.172201 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.172305 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.173087 (+ 782us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.173407 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015}
I20260620 22:54:17.175112 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.161244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.2.90.189:38383)
0620 22:54:17.161276 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.161278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.161285 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:54:17.161302 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.161303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.161312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.161386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.162043 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.162690 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.163653 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.163916 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.163979 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.172430 (+ 8451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.172432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.172432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.172441 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.173974 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.174323 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.174325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.174325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.174389 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.174580 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.174581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.174581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.174714 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.174820 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.174948 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.175022 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:54:17.179363 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.170306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.2.90.188:35349)
0620 22:54:17.173559 (+ 3253us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.173561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.173570 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:17.173589 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.173589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.173599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.173923 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.173949 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.174401 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.175287 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.175560 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.175612 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.177372 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.177373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.177373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.177379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.178496 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.178825 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.178825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.178826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.178868 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.179011 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.179011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.179011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.179093 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.179134 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.179262 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.179310 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3239,"thread_start_us":71,"threads_started":1}
I20260620 22:54:17.179426 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.170062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42580)
0620 22:54:17.170141 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.170150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:17.170182 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:17.173683 (+ 3501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:17.173684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:17.173829 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:17.173866 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.173867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.174470 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.174471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.175213 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.175216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.175623 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.175624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.175678 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.175995 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:17.175999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.177295 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.178565 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.178566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.178567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.178764 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.178908 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.178908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.178909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.178962 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.179140 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.179141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.179180 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.179246 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.179325 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 22:54:17.180958 15748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: Bootstrap starting.
I20260620 22:54:17.181669 15748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:17.181957 15748 log.cc:824] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:17.182689 15748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: No bootstrap required, opened a new log
I20260620 22:54:17.184036 15748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.184146 15748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:17.184177 15748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eda3e3daf540288980ac57958bd471, State: Initialized, Role: FOLLOWER
I20260620 22:54:17.184293 15748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.184566 15748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:17.184530 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.184612 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:54:17.186098 15937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:54:17.186196 15937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:54:17.187224 15680 master_runner.cc:186] Error getting master registration for 127.2.90.189:38383: OK, instance_id { permanent_uuid: "11d0f122dc654073882331dd391d1c44" instance_seqno: 1781996056983243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:54:17.189435 15680 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:54:17.214826 15823 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } attempt: 3
Jun 20 22:54:17 dist-test-slave-4mtw krb5kdc[15667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:54:17.222787 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.215196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.2.90.188:35349)
0620 22:54:17.215237 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:17.215239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:17.215248 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:17.215312 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:17.215313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:17.215322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:17.215378 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:17.215679 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.216450 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.217492 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.217797 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.217854 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.220157 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:17.220159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:17.220159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:17.220166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:17.221675 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.222048 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.222049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.222050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.222108 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:17.222288 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:17.222288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:17.222289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:17.222424 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:17.222486 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:17.222627 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 22:54:17.222701 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:54:17.222805 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:17.215112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:35349 (local address 127.0.0.1:42590)
0620 22:54:17.215184 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:17.215195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:17.215241 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:17.215380 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:17.215380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:17.215562 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:17.215614 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.215616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.216519 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.216521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.217363 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:17.217366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:17.217962 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:17.217963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:17.218022 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:17.218586 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:17.218592 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:17.220081 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:17.221754 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.221755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.221756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.221969 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.222168 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:17.222169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:17.222170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:17.222232 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:17.222483 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:17.222484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:17.222533 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:54:17.222601 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:54:17.222699 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 22:54:17.224885 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: Bootstrap starting.
I20260620 22:54:17.225596 15823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:17.225905 15823 log.cc:824] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:17.226991 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: No bootstrap required, opened a new log
I20260620 22:54:17.228929 15823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.229094 15823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:17.229130 15823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d0f122dc654073882331dd391d1c44, State: Initialized, Role: FOLLOWER
I20260620 22:54:17.229249 15823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.229553 15823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 22:54:17.231228 15776 tablet.cc:2369] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:54:17.231393 15951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:54:17.231508 15951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:54:17.233999 15755 master_runner.cc:263] No leader master found from master 11d0f122dc654073882331dd391d1c44
I20260620 22:54:17.234050 15755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:54:17.229511 15940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.234110 15940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:54:17.317394 15911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:54:17.317526 15911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.317771 15911 leader_election.cc:288] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47eda3e3daf540288980ac57958bd471 (127.2.90.190:42239), 11d0f122dc654073882331dd391d1c44 (127.2.90.189:38383)
I20260620 22:54:17.318011 15722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40627acda2c8463c8b14137ef44acc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eda3e3daf540288980ac57958bd471" is_pre_election: true
I20260620 22:54:17.318064 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40627acda2c8463c8b14137ef44acc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f122dc654073882331dd391d1c44" is_pre_election: true
I20260620 22:54:17.318173 15797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40627acda2c8463c8b14137ef44acc59 in term 0.
I20260620 22:54:17.318174 15722 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40627acda2c8463c8b14137ef44acc59 in term 0.
I20260620 22:54:17.318343 15846 leader_election.cc:302] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f122dc654073882331dd391d1c44, 40627acda2c8463c8b14137ef44acc59; no voters:
I20260620 22:54:17.318464 15911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:54:17.318521 15911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:54:17.318542 15911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:17.319180 15911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.319293 15911 leader_election.cc:288] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [CANDIDATE]: Term 1 election: Requested vote from peers 47eda3e3daf540288980ac57958bd471 (127.2.90.190:42239), 11d0f122dc654073882331dd391d1c44 (127.2.90.189:38383)
I20260620 22:54:17.319451 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40627acda2c8463c8b14137ef44acc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d0f122dc654073882331dd391d1c44"
I20260620 22:54:17.319491 15722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40627acda2c8463c8b14137ef44acc59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eda3e3daf540288980ac57958bd471"
I20260620 22:54:17.319531 15797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:17.319552 15722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:17.320314 15722 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40627acda2c8463c8b14137ef44acc59 in term 1.
I20260620 22:54:17.320314 15797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40627acda2c8463c8b14137ef44acc59 in term 1.
I20260620 22:54:17.320459 15846 leader_election.cc:302] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d0f122dc654073882331dd391d1c44, 40627acda2c8463c8b14137ef44acc59; no voters:
I20260620 22:54:17.320544 15911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:17.320708 15911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [term 1 LEADER]: Becoming Leader. State: Replica: 40627acda2c8463c8b14137ef44acc59, State: Running, Role: LEADER
I20260620 22:54:17.320832 15911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } }
I20260620 22:54:17.321215 15953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40627acda2c8463c8b14137ef44acc59. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.321293 15953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:17.321463 15955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:17.321676 15955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:17.322901 15797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [term 1 FOLLOWER]: Refusing update from remote peer 40627acda2c8463c8b14137ef44acc59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:17.322965 15722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [term 1 FOLLOWER]: Refusing update from remote peer 40627acda2c8463c8b14137ef44acc59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:17.323253 15953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:17.323465 15953 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:17.324514 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40627acda2c8463c8b14137ef44acc59. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.324514 15940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40627acda2c8463c8b14137ef44acc59. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.324597 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:54:17.324606 15940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:54:17.325834 15956 mvcc.cc:205] Tried to move back new op lower bound from 7299055850792919040 to 7299055850786762752. Current Snapshot: MvccSnapshot[applied={T|T < 7299055850792919040}]
I20260620 22:54:17.325989 15953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.326064 15953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:17.326354 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.326426 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:54:17.326427 15955 catalog_manager.cc:1347] Generated new cluster ID: c0de7e1e7b934dc7a5cc7c3ed193cafb
I20260620 22:54:17.326457 15955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:54:17.326643 15958 mvcc.cc:205] Tried to move back new op lower bound from 7299055850792919040 to 7299055850786762752. Current Snapshot: MvccSnapshot[applied={T|T < 7299055850792919040}]
I20260620 22:54:17.329697 15953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.329777 15953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:17.330310 15940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40627acda2c8463c8b14137ef44acc59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47eda3e3daf540288980ac57958bd471" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42239 } } peers { permanent_uuid: "11d0f122dc654073882331dd391d1c44" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 38383 } } peers { permanent_uuid: "40627acda2c8463c8b14137ef44acc59" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 35349 } } }
I20260620 22:54:17.330386 15940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:54:17.348143 15955 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:54:17.348662 15955 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:17.353503 15955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40627acda2c8463c8b14137ef44acc59: Generated new TSK 0
I20260620 22:54:17.353674 15955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:54:18.186607 15937 catalog_manager.cc:1259] Loaded cluster ID: c0de7e1e7b934dc7a5cc7c3ed193cafb
I20260620 22:54:18.186681 15937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: loading cluster ID for follower catalog manager: success
I20260620 22:54:18.187456 15937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: acquiring CA information for follower catalog manager: success
I20260620 22:54:18.187747 15937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47eda3e3daf540288980ac57958bd471: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:54:18.231952 15951 catalog_manager.cc:1259] Loaded cluster ID: c0de7e1e7b934dc7a5cc7c3ed193cafb
I20260620 22:54:18.232030 15951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: loading cluster ID for follower catalog manager: success
I20260620 22:54:18.232900 15951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: acquiring CA information for follower catalog manager: success
I20260620 22:54:18.233196 15951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11d0f122dc654073882331dd391d1c44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:54:18.404294 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15680
I20260620 22:54:18.409040 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15755
I20260620 22:54:18.412981 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15830
2026-06-20T22:54:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2752 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:54:18 dist-test-slave-4mtw krb5kdc[15970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:54:18 dist-test-slave-4mtw krb5kdc[15970](info): set up 2 sockets
Jun 20 22:54:18 dist-test-slave-4mtw krb5kdc[15970](info): commencing operation
krb5kdc: starting...
W20260620 22:54:19.352399 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:54:19 dist-test-slave-4mtw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:54:19.456728 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:40801
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:19.533231 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:19.533423 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:19.533452 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:19.534574 15983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:19.534616 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:19.534636 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:19.534651 15983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:19.534665 15983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:19.536092 15983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:40801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:19.536320 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:19.536512 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:19.540884 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:19.540891 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:19.540891 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:19.541090 15983 server_base.cc:1034] running on GCE node
I20260620 22:54:19.541326 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:19.541605 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:19.542743 15983 hybrid_clock.cc:648] HybridClock initialized: now 1781996059542727 us; error 27 us; skew 500 ppm
Jun 20 22:54:19 dist-test-slave-4mtw krb5kdc[15970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996059, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:19.545006 15983 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:19.545406 15983 webserver.cc:460] Webserver started at http://127.2.90.190:34633/ using document root <none> and password file <none>
I20260620 22:54:19.545615 15983 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:19.545661 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:19.545780 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:19.546727 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "311af6db4eee45ba923f1c40dc0cac90"
format_stamp: "Formatted at 2026-06-20 22:54:19 on dist-test-slave-4mtw"
server_key: "a30424569e9b840f340bd6029117c419"
server_key_iv: "7555a44f80cae5e1c1d8d12d51a4a695"
server_key_version: "encryptionkey@0"
I20260620 22:54:19.547036 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "311af6db4eee45ba923f1c40dc0cac90"
format_stamp: "Formatted at 2026-06-20 22:54:19 on dist-test-slave-4mtw"
server_key: "a30424569e9b840f340bd6029117c419"
server_key_iv: "7555a44f80cae5e1c1d8d12d51a4a695"
server_key_version: "encryptionkey@0"
I20260620 22:54:19.548173 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:54:19.548828 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:19.548980 15983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:19.549052 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "311af6db4eee45ba923f1c40dc0cac90"
format_stamp: "Formatted at 2026-06-20 22:54:19 on dist-test-slave-4mtw"
server_key: "a30424569e9b840f340bd6029117c419"
server_key_iv: "7555a44f80cae5e1c1d8d12d51a4a695"
server_key_version: "encryptionkey@0"
I20260620 22:54:19.549116 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:19.574270 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:19.576397 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:19.576545 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:19.580186 15983 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:40801
I20260620 22:54:19.580222 16050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:40801 every 8 connection(s)
I20260620 22:54:19.580585 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:19.581393 16051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:19.583535 16051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90: Bootstrap starting.
I20260620 22:54:19.584031 16051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:19.584244 16051 log.cc:824] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:19.584889 16051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90: No bootstrap required, opened a new log
I20260620 22:54:19.586374 16051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311af6db4eee45ba923f1c40dc0cac90" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40801 } }
I20260620 22:54:19.586519 16051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:19.586540 16051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311af6db4eee45ba923f1c40dc0cac90, State: Initialized, Role: FOLLOWER
I20260620 22:54:19.586645 16051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311af6db4eee45ba923f1c40dc0cac90" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40801 } }
I20260620 22:54:19.586690 16051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:19.586709 16051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:19.586735 16051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:19.587178 16051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311af6db4eee45ba923f1c40dc0cac90" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40801 } }
I20260620 22:54:19.587255 16051 leader_election.cc:302] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 311af6db4eee45ba923f1c40dc0cac90; no voters:
I20260620 22:54:19.587380 16051 leader_election.cc:288] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:54:19.587428 16054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:19.587587 16054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [term 1 LEADER]: Becoming Leader. State: Replica: 311af6db4eee45ba923f1c40dc0cac90, State: Running, Role: LEADER
I20260620 22:54:19.587592 16051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:19.587729 16054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311af6db4eee45ba923f1c40dc0cac90" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40801 } }
I20260620 22:54:19.588245 16055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "311af6db4eee45ba923f1c40dc0cac90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311af6db4eee45ba923f1c40dc0cac90" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40801 } } }
I20260620 22:54:19.588326 16055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:19.588641 16056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311af6db4eee45ba923f1c40dc0cac90. Latest consensus state: current_term: 1 leader_uuid: "311af6db4eee45ba923f1c40dc0cac90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "311af6db4eee45ba923f1c40dc0cac90" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 40801 } } }
I20260620 22:54:19.588719 16063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:19.588765 16056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:19.589149 16063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:19.590497 16063 catalog_manager.cc:1347] Generated new cluster ID: c7bd6e1a00e845c09d7208e4788f799f
I20260620 22:54:19.590536 16063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:54:19.590600 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 15983
I20260620 22:54:19.590701 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:54:19.590884 2410 external_mini_cluster.cc:1428] Setting key 892e0e7cb4b1ae251e21fc28bb3dee33
Jun 20 22:54:19 dist-test-slave-4mtw krb5kdc[15970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:19.601644 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:19.591291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.2.90.190:40801)
0620 22:54:19.591441 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:19.591450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:19.591486 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:54:19.591623 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:19.591630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:19.591678 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:19.591801 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:19.592564 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:19.593937 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:19.594995 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:19.595588 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:19.595676 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:19.598287 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:19.598291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:19.598291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:19.598302 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:19.600123 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:19.600458 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:19.600461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:19.600461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:19.600581 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:19.600760 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:19.600761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:19.600761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:19.601053 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:19.601217 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:19.601281 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:54:19.601409 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260620 22:54:19.602567 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:54:19.617981 16063 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:54:19.618664 16063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:19.625612 16063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311af6db4eee45ba923f1c40dc0cac90: Generated new TSK 0
I20260620 22:54:19.625770 16063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:54:19.656116 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 15983
2026-06-20T22:54:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1242 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:54:19 dist-test-slave-4mtw krb5kdc[16078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:54:19 dist-test-slave-4mtw krb5kdc[16078](info): set up 2 sockets
Jun 20 22:54:19 dist-test-slave-4mtw krb5kdc[16078](info): commencing operation
krb5kdc: starting...
W20260620 22:54:20.596415 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:54:20 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:54:20.700234 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44667
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:20.778429 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:20.778627 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:20.778647 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:20.779750 16091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:20.779788 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:20.779801 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:20.779825 16091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:20.779847 16091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:20.781322 16091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:20.781535 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:20.781726 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:20.786247 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:20.786268 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:20.786275 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:20.786350 16091 server_base.cc:1034] running on GCE node
I20260620 22:54:20.786679 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:20.786963 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:20.788100 16091 hybrid_clock.cc:648] HybridClock initialized: now 1781996060788086 us; error 29 us; skew 500 ppm
Jun 20 22:54:20 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:20.790264 16091 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:20.790637 16091 webserver.cc:460] Webserver started at http://127.2.90.190:39769/ using document root <none> and password file <none>
I20260620 22:54:20.790820 16091 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:20.790865 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:20.790982 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:20.791934 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "cebed98135a24b01b5051dc528b8d1e7"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7926bc036b95582b358d032c7c3e1416"
server_key_iv: "c3412a50a3fe11aa0d9e2b3cfefb4959"
server_key_version: "encryptionkey@0"
I20260620 22:54:20.792254 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "cebed98135a24b01b5051dc528b8d1e7"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7926bc036b95582b358d032c7c3e1416"
server_key_iv: "c3412a50a3fe11aa0d9e2b3cfefb4959"
server_key_version: "encryptionkey@0"
I20260620 22:54:20.793428 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:54:20.794064 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:20.794199 16091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:20.794276 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "cebed98135a24b01b5051dc528b8d1e7"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7926bc036b95582b358d032c7c3e1416"
server_key_iv: "c3412a50a3fe11aa0d9e2b3cfefb4959"
server_key_version: "encryptionkey@0"
I20260620 22:54:20.794332 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:20.802198 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:20.803648 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:20.803772 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:20.807297 16091 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:44667
I20260620 22:54:20.807325 16158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:44667 every 8 connection(s)
I20260620 22:54:20.807600 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:20.808446 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:20.810495 16159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap starting.
I20260620 22:54:20.811010 16159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:20.811244 16159 log.cc:824] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:20.811827 16159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: No bootstrap required, opened a new log
I20260620 22:54:20.813385 16159 raft_consensus.cc:348] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:20.813509 16159 raft_consensus.cc:374] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:20.813531 16159 raft_consensus.cc:729] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebed98135a24b01b5051dc528b8d1e7, State: Initialized, Role: FOLLOWER
I20260620 22:54:20.813656 16159 consensus_queue.cc:260] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:20.813724 16159 raft_consensus.cc:388] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:20.813757 16159 raft_consensus.cc:482] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:20.813802 16159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:20.814119 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16091
I20260620 22:54:20.814209 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:54:20.814368 2410 external_mini_cluster.cc:1428] Setting key 530c962941bf72011fa7290656143e3c
I20260620 22:54:20.814596 16159 raft_consensus.cc:504] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:20.814689 16159 leader_election.cc:302] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cebed98135a24b01b5051dc528b8d1e7; no voters:
I20260620 22:54:20.814872 16159 leader_election.cc:288] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:54:20.814961 16164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:20.815109 16159 sys_catalog.cc:564] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:20.815135 16164 raft_consensus.cc:686] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 LEADER]: Becoming Leader. State: Replica: cebed98135a24b01b5051dc528b8d1e7, State: Running, Role: LEADER
I20260620 22:54:20.816233 16164 consensus_queue.cc:237] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:20.816988 16165 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cebed98135a24b01b5051dc528b8d1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } } }
I20260620 22:54:20.817034 16165 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: This master's current role is: LEADER
W20260620 22:54:20.817194 16178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 22:54:20.816828 16166 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cebed98135a24b01b5051dc528b8d1e7. Latest consensus state: current_term: 1 leader_uuid: "cebed98135a24b01b5051dc528b8d1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } } }
W20260620 22:54:20.817261 16178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:54:20.817272 16166 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:20.817394 16180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:20.817536 16180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:20.818727 16180 catalog_manager.cc:1347] Generated new cluster ID: 05aeb5716b634dca8e6fbd0a46c1bbf4
I20260620 22:54:20.818773 16180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:54:20 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:20.825670 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:20.814744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.2.90.190:44667)
0620 22:54:20.814899 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:20.814909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:20.814945 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:54:20.815055 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:20.815062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:20.815120 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:20.815307 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:20.816000 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:20.817439 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:20.818482 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:20.819052 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:20.819145 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:20.821711 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:20.821715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:20.821716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:20.821727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:20.824041 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:20.824379 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:20.824382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:20.824383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:20.824499 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:20.824668 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:20.824669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:20.824670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:20.824979 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:20.825128 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:20.825208 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:54:20.825456 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260620 22:54:20.830482 16180 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:54:20.830914 16180 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:20.836285 16180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Generated new TSK 0
I20260620 22:54:20.836437 16180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:54:20.865097 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44667
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:20.942775 16185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:20.942981 16185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:20.943004 16185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:20.944094 16185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:20.944132 16185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:20.944156 16185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:20.945566 16185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:20.945765 16185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:20.945943 16185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:20.950546 16193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:20.950552 16190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:20.950577 16191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:20.950752 16185 server_base.cc:1034] running on GCE node
I20260620 22:54:20.950955 16185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:20.951187 16185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:20.952330 16185 hybrid_clock.cc:648] HybridClock initialized: now 1781996060952314 us; error 31 us; skew 500 ppm
Jun 20 22:54:20 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:20.954610 16185 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:20.955010 16185 webserver.cc:460] Webserver started at http://127.2.90.129:46409/ using document root <none> and password file <none>
I20260620 22:54:20.955212 16185 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:20.955258 16185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:20.955380 16185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:20.956387 16185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "56bad906eb00437b85726b3409c40eba"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7c1aa85de3cd8444637497a6c583f4fc"
server_key_iv: "e84050f04326753c2b6f61a7491e3e93"
server_key_version: "encryptionkey@0"
I20260620 22:54:20.956772 16185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "56bad906eb00437b85726b3409c40eba"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7c1aa85de3cd8444637497a6c583f4fc"
server_key_iv: "e84050f04326753c2b6f61a7491e3e93"
server_key_version: "encryptionkey@0"
I20260620 22:54:20.957965 16185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:54:20.958601 16200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:20.958747 16185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:54:20.958827 16185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "56bad906eb00437b85726b3409c40eba"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7c1aa85de3cd8444637497a6c583f4fc"
server_key_iv: "e84050f04326753c2b6f61a7491e3e93"
server_key_version: "encryptionkey@0"
I20260620 22:54:20.958896 16185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:20.971872 16185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:20.973783 16185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:20.973908 16185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:20.974128 16185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:20.974429 16185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:20.974462 16185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:20.974490 16185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:20.974507 16185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:20.979996 16185 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37147
I20260620 22:54:20.980047 16313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37147 every 8 connection(s)
I20260620 22:54:20.980340 16185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:54:20 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996060, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:20.988298 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:20.981011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:41325 (local address 127.2.90.190:44667)
0620 22:54:20.981086 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:20.981089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:20.981243 (+ 154us) server_negotiation.cc:407] Connection header received
0620 22:54:20.981717 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:20.981718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:20.981731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:20.981770 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:20.982629 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:20.983125 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:20.984057 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:20.984239 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:20.984280 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:20.986208 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:20.986209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:20.986209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:20.986214 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:20.987295 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:20.987620 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:20.987620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:20.987621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:20.987665 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:20.987845 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:20.987845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:20.987845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:20.987933 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:20.987985 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:20.988172 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 22:54:20.988234 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 22:54:20.988456 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:20.980984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:41325)
0620 22:54:20.981131 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:20.981168 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:20.981629 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:20.981834 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:20.981837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:20.982367 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:20.982555 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:20.982559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:20.983195 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:20.983196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:20.983998 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:20.984001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:20.984290 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:20.984291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:20.984387 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:20.984861 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:20.984869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:20.986137 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:20.987358 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:20.987364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:20.987366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:20.987566 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:20.987706 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:20.987707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:20.987707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:20.987804 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:20.987972 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:20.987975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:20.988076 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 22:54:20.988153 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 22:54:20.988273 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260620 22:54:20.988917 16314 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44667
I20260620 22:54:20.989002 16314 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:20.989140 16314 heartbeater.cc:511] Master 127.2.90.190:44667 requested a full tablet report, sending...
I20260620 22:54:20.989351 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16185
I20260620 22:54:20.989449 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:54:20.989642 2410 external_mini_cluster.cc:1428] Setting key 56308277c9e7ae6e495ebd8cefa9ded6
I20260620 22:54:20.989657 16123 ts_manager.cc:194] Registered new tserver with Master: 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147)
I20260620 22:54:20.990412 16123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:41325
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:54:21.011112 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44667
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:21.087723 16319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:21.087903 16319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:21.087924 16319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:21.089035 16319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:21.089071 16319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:21.089097 16319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:54:21.090446 16319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:21.090620 16319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:21.090787 16319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:21.095373 16327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.095461 16319 server_base.cc:1034] running on GCE node
W20260620 22:54:21.095396 16325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:21.095386 16324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.095781 16319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:21.095984 16319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:21.097113 16319 hybrid_clock.cc:648] HybridClock initialized: now 1781996061097100 us; error 28 us; skew 500 ppm
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:21.099303 16319 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:54:21.099704 16319 webserver.cc:460] Webserver started at http://127.2.90.130:40843/ using document root <none> and password file <none>
I20260620 22:54:21.099900 16319 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:21.099944 16319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:21.100057 16319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:21.101055 16319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "d2cc2396ef084bba93b850f18a6314ba"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "dbf5fdd8c279134ee96f713da5f0e1ab"
server_key_iv: "605e02ac87960612ab75ad8cafa6b10f"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.101359 16319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "d2cc2396ef084bba93b850f18a6314ba"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "dbf5fdd8c279134ee96f713da5f0e1ab"
server_key_iv: "605e02ac87960612ab75ad8cafa6b10f"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.102553 16319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:54:21.103135 16334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.103268 16319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:21.103343 16319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "d2cc2396ef084bba93b850f18a6314ba"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "dbf5fdd8c279134ee96f713da5f0e1ab"
server_key_iv: "605e02ac87960612ab75ad8cafa6b10f"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.103399 16319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:21.109768 16319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:21.111258 16319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:21.111377 16319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:21.111559 16319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:21.111821 16319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:21.111853 16319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.111879 16319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:21.111902 16319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.117235 16319 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45007
I20260620 22:54:21.117266 16447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45007 every 8 connection(s)
I20260620 22:54:21.117550 16319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:21.125231 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.118141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:33395 (local address 127.2.90.190:44667)
0620 22:54:21.118182 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.118184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.118359 (+ 175us) server_negotiation.cc:407] Connection header received
0620 22:54:21.118791 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.118792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.118803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.118848 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:21.119652 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.120140 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.121080 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.121261 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.121308 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.123025 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:21.123026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:21.123027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:21.123033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:21.124238 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.124530 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.124531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.124531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.124577 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.124771 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.124771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.124771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.124866 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:21.124913 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.125102 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.125168 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:54:21.125370 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16319
I20260620 22:54:21.125394 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.118117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:33395)
0620 22:54:21.118250 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.118282 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.118718 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.118899 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.118901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.119406 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:21.119588 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.119593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.120200 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.120200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.121018 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.121021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.121313 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.121314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.121399 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.121772 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:21.121779 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:21.122948 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:21.124294 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.124296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.124297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.124481 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.124619 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.124619 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.124620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.124730 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.124909 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:21.124912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:21.125013 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.125085 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.125190 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260620 22:54:21.125476 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:54:21.125661 2410 external_mini_cluster.cc:1428] Setting key f1dfd7f2e8533964c3455b178fdacb81
I20260620 22:54:21.125917 16448 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44667
I20260620 22:54:21.126024 16448 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:21.126187 16448 heartbeater.cc:511] Master 127.2.90.190:44667 requested a full tablet report, sending...
I20260620 22:54:21.126556 16123 ts_manager.cc:194] Registered new tserver with Master: d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007)
I20260620 22:54:21.127000 16123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:33395
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:54:21.146754 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44667
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:21.228484 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:21.228713 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:21.228749 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:21.230014 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:21.230072 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:21.230111 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:54:21.231727 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:21.231956 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:21.232162 16453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:21.236846 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:21.236865 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.236925 16453 server_base.cc:1034] running on GCE node
W20260620 22:54:21.236847 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.237262 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:21.237493 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:21.238628 16453 hybrid_clock.cc:648] HybridClock initialized: now 1781996061238617 us; error 26 us; skew 500 ppm
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:21.240885 16453 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:54:21.241277 16453 webserver.cc:460] Webserver started at http://127.2.90.131:38869/ using document root <none> and password file <none>
I20260620 22:54:21.241469 16453 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:21.241516 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:21.241632 16453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:21.242619 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "d619ce7db73e407988ee8112e2543ed4"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "096773c66d7d5a584e916e99708c8b3f"
server_key_iv: "786cc823b7892725fe66701de9c3a887"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.242935 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "d619ce7db73e407988ee8112e2543ed4"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "096773c66d7d5a584e916e99708c8b3f"
server_key_iv: "786cc823b7892725fe66701de9c3a887"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.244069 16453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:54:21.244719 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.244858 16453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.244943 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d619ce7db73e407988ee8112e2543ed4"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "096773c66d7d5a584e916e99708c8b3f"
server_key_iv: "786cc823b7892725fe66701de9c3a887"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.245005 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:21.258047 16453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:21.259619 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:21.259742 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:21.259939 16453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:21.260214 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:21.260246 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.260272 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:21.260288 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.265617 16453 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42853
I20260620 22:54:21.265653 16581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42853 every 8 connection(s)
I20260620 22:54:21.265941 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:54:21.270800 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16453
I20260620 22:54:21.270900 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:54:21.271072 2410 external_mini_cluster.cc:1428] Setting key 234d59ec4757707264bb44b35aa6a115
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:21.273909 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.266584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:36543 (local address 127.2.90.190:44667)
0620 22:54:21.266637 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.266638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.266823 (+ 185us) server_negotiation.cc:407] Connection header received
0620 22:54:21.267276 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.267277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.267285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.267339 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:21.268167 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.268650 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.269592 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.269772 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.269819 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.271736 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:21.271737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:21.271737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:21.271744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:21.272906 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.273242 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.273243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.273243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.273293 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.273468 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.273468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.273469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.273565 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:21.273608 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.273794 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.273848 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:54:21.274084 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.266563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:36543)
0620 22:54:21.266708 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.266746 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.267203 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.267391 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.267394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.267912 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:21.268102 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.268107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.268754 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.268755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.269528 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.269530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.269826 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.269827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.269917 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.270317 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:21.270324 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:21.271663 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:21.272968 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.272973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.272974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.273188 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.273335 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.273336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.273336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.273419 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.273604 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:21.273607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:21.273708 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.273781 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.273887 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260620 22:54:21.274452 16582 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44667
I20260620 22:54:21.274551 16582 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:21.274688 16582 heartbeater.cc:511] Master 127.2.90.190:44667 requested a full tablet report, sending...
I20260620 22:54:21.275004 16123 ts_manager.cc:194] Registered new tserver with Master: d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853)
I20260620 22:54:21.275336 16123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:36543
I20260620 22:54:21.282600 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:54:21.289258 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.283798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.2.90.190:44667)
0620 22:54:21.283852 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.283854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.283975 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:54:21.284044 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.284044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.284054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.284097 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:21.284596 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.285111 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.285823 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.286109 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.286188 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.287237 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:21.287238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:21.287238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:21.287245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:21.288228 (+ 983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.288523 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.288524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.288525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.288578 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.288758 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.288759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.288759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.288901 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:21.288984 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.289103 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.289172 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:54:21.290467 16123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:54:21.291949 16123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:54:21.297946 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.293928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37147 (local address 127.0.0.1:40228)
0620 22:54:21.294144 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.294154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.294204 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.294633 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.294634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.294636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:21.294684 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.294686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.295630 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.295631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.296755 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.296757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.297669 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.297670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.297745 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.297756 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.297811 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.297861 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"spinlock_wait_cycles":384,"thread_start_us":121,"threads_started":1}
I20260620 22:54:21.297946 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.294310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.2.90.129:37147)
0620 22:54:21.294460 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.294463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.294472 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:21.294508 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.294511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.294525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.294632 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:21.294833 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.295541 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.296841 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.297588 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.297649 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.297710 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.297831 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.297871 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260620 22:54:21.298250 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.294104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.2.90.131:42853)
0620 22:54:21.294331 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.294337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.294346 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:21.294410 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.294412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.294427 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.294509 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:21.294766 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.295869 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.296984 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.297664 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.297728 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.297781 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.298129 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.298174 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260620 22:54:21.298310 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.293928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45007 (local address 127.0.0.1:60010)
0620 22:54:21.294484 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.294493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.294535 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.294835 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.294836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.294838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:21.294871 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.294872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.296069 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.296070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.297147 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.297150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.298113 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.298114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.298168 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.298171 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.298224 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.298257 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":3,"thread_start_us":166,"threads_started":1}
I20260620 22:54:21.298381 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.294176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.2.90.130:45007)
0620 22:54:21.294630 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.294634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.294642 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:54:21.294679 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.294681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.294694 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.294759 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:21.294941 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.295978 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.297222 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.297977 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.298033 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.298082 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.298255 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.298307 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":88,"threads_started":1}
I20260620 22:54:21.298065 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.293922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42853 (local address 127.0.0.1:44730)
0620 22:54:21.294100 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.294139 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.294201 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.294515 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.294517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.294520 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:21.294585 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.294587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.295976 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.295977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.296917 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.296920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.297843 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.297844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.297904 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.297908 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.297955 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.297993 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":35,"thread_start_us":53,"threads_started":1}
I20260620 22:54:21.299003 16382 tablet_service.cc:1467] Processing CreateTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa (DEFAULT_TABLE table=test-table [id=8df589a315fa428ab1afcb92e1a47952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:21.298956 16248 tablet_service.cc:1467] Processing CreateTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa (DEFAULT_TABLE table=test-table [id=8df589a315fa428ab1afcb92e1a47952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:21.299190 16516 tablet_service.cc:1467] Processing CreateTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa (DEFAULT_TABLE table=test-table [id=8df589a315fa428ab1afcb92e1a47952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:21.299312 16382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949af1efc6404801bd4d4d2cb43ad7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:21.299312 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949af1efc6404801bd4d4d2cb43ad7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:21.299422 16516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949af1efc6404801bd4d4d2cb43ad7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:21.301869 16596 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Bootstrap starting.
I20260620 22:54:21.302412 16598 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Bootstrap starting.
I20260620 22:54:21.302677 16596 tablet_bootstrap.cc:654] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:21.303028 16598 tablet_bootstrap.cc:654] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:21.303260 16596 log.cc:824] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:21.303550 16598 log.cc:824] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:21.303859 16597 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Bootstrap starting.
I20260620 22:54:21.304009 16596 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: No bootstrap required, opened a new log
I20260620 22:54:21.304076 16596 ts_tablet_manager.cc:1397] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:54:21.304395 16597 tablet_bootstrap.cc:654] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:21.304876 16597 log.cc:824] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:21.305343 16598 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: No bootstrap required, opened a new log
I20260620 22:54:21.305403 16598 ts_tablet_manager.cc:1397] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:54:21.305554 16597 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: No bootstrap required, opened a new log
I20260620 22:54:21.305625 16597 ts_tablet_manager.cc:1397] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:54:21.305696 16596 raft_consensus.cc:348] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.305873 16596 raft_consensus.cc:374] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:21.305939 16596 raft_consensus.cc:729] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d619ce7db73e407988ee8112e2543ed4, State: Initialized, Role: FOLLOWER
I20260620 22:54:21.306059 16596 consensus_queue.cc:260] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.306316 16596 ts_tablet_manager.cc:1428] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:54:21.306430 16582 heartbeater.cc:503] Master 127.2.90.190:44667 was elected leader, sending a full tablet report...
I20260620 22:54:21.307089 16598 raft_consensus.cc:348] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.307307 16598 raft_consensus.cc:374] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:21.307360 16598 raft_consensus.cc:729] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cc2396ef084bba93b850f18a6314ba, State: Initialized, Role: FOLLOWER
I20260620 22:54:21.307497 16598 consensus_queue.cc:260] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.307516 16597 raft_consensus.cc:348] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.307700 16597 raft_consensus.cc:374] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:21.307751 16597 raft_consensus.cc:729] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bad906eb00437b85726b3409c40eba, State: Initialized, Role: FOLLOWER
I20260620 22:54:21.307760 16598 ts_tablet_manager.cc:1428] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:54:21.307860 16448 heartbeater.cc:503] Master 127.2.90.190:44667 was elected leader, sending a full tablet report...
I20260620 22:54:21.307860 16597 consensus_queue.cc:260] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.308107 16597 ts_tablet_manager.cc:1428] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:54:21.308140 16314 heartbeater.cc:503] Master 127.2.90.190:44667 was elected leader, sending a full tablet report...
W20260620 22:54:21.368041 16449 tablet.cc:2369] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:54:21.481029 16315 tablet.cc:2369] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:54:21.499518 16604 raft_consensus.cc:482] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:54:21.499643 16604 raft_consensus.cc:504] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.499974 16604 leader_election.cc:288] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853), d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007)
I20260620 22:54:21.503847 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.500227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:51499 (local address 127.2.90.131:42853)
0620 22:54:21.500279 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.500281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.500350 (+ 69us) server_negotiation.cc:407] Connection header received
0620 22:54:21.500402 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.500403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.500412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.500478 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:21.500646 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.501401 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.502932 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.503516 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.503580 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.503587 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.503740 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.503777 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:54:21.503847 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.500246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:55911 (local address 127.2.90.130:45007)
0620 22:54:21.500292 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.500294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.500347 (+ 53us) server_negotiation.cc:407] Connection header received
0620 22:54:21.500402 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.500403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.500414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.500472 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:21.500651 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.501401 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.502937 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.503508 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.503577 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.503586 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.503740 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.503777 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:54:21.503902 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.500142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42853 (local address 127.2.90.129:51499)
0620 22:54:21.500292 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.500301 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.500350 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.500487 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.500488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.500489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:21.500583 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.500586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.501483 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.501483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.502852 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.502856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.503587 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.503588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.503674 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.503680 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.503731 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.503784 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":83,"thread_start_us":62,"threads_started":1}
I20260620 22:54:21.503904 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.500141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45007 (local address 127.2.90.129:55911)
0620 22:54:21.500265 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.500277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.500347 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.500486 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.500487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.500489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:21.500593 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.500594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.501474 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.501475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.502852 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.502856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.503589 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.503589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.503674 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.503681 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.503731 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.503784 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":90,"thread_start_us":70,"threads_started":1}
I20260620 22:54:21.504205 16536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949af1efc6404801bd4d4d2cb43ad7fa" candidate_uuid: "56bad906eb00437b85726b3409c40eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d619ce7db73e407988ee8112e2543ed4" is_pre_election: true
I20260620 22:54:21.504371 16536 raft_consensus.cc:2454] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56bad906eb00437b85726b3409c40eba in term 0.
I20260620 22:54:21.504377 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949af1efc6404801bd4d4d2cb43ad7fa" candidate_uuid: "56bad906eb00437b85726b3409c40eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cc2396ef084bba93b850f18a6314ba" is_pre_election: true
I20260620 22:54:21.504527 16402 raft_consensus.cc:2454] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56bad906eb00437b85726b3409c40eba in term 0.
I20260620 22:54:21.504592 16204 leader_election.cc:302] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56bad906eb00437b85726b3409c40eba, d619ce7db73e407988ee8112e2543ed4; no voters:
I20260620 22:54:21.504727 16604 raft_consensus.cc:2781] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:54:21.504777 16604 raft_consensus.cc:482] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:54:21.504798 16604 raft_consensus.cc:3037] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:21.505472 16604 raft_consensus.cc:504] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.505597 16604 leader_election.cc:288] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 1 election: Requested vote from peers d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853), d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007)
I20260620 22:54:21.505736 16536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949af1efc6404801bd4d4d2cb43ad7fa" candidate_uuid: "56bad906eb00437b85726b3409c40eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d619ce7db73e407988ee8112e2543ed4"
I20260620 22:54:21.505741 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949af1efc6404801bd4d4d2cb43ad7fa" candidate_uuid: "56bad906eb00437b85726b3409c40eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cc2396ef084bba93b850f18a6314ba"
I20260620 22:54:21.505808 16536 raft_consensus.cc:3037] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:21.505820 16402 raft_consensus.cc:3037] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:21.506438 16402 raft_consensus.cc:2454] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56bad906eb00437b85726b3409c40eba in term 1.
I20260620 22:54:21.506495 16536 raft_consensus.cc:2454] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56bad906eb00437b85726b3409c40eba in term 1.
I20260620 22:54:21.506655 16204 leader_election.cc:302] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56bad906eb00437b85726b3409c40eba, d619ce7db73e407988ee8112e2543ed4; no voters:
I20260620 22:54:21.506734 16604 raft_consensus.cc:2781] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:21.506851 16604 raft_consensus.cc:686] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 LEADER]: Becoming Leader. State: Replica: 56bad906eb00437b85726b3409c40eba, State: Running, Role: LEADER
I20260620 22:54:21.506940 16604 consensus_queue.cc:237] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.507699 16123 catalog_manager.cc:5495] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba reported cstate change: term changed from 0 to 1, leader changed from <none> to 56bad906eb00437b85726b3409c40eba (127.2.90.129). New cstate: current_term: 1 leader_uuid: "56bad906eb00437b85726b3409c40eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } health_report { overall_health: HEALTHY } } }
W20260620 22:54:21.516376 16583 tablet.cc:2369] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:54:21.522262 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.520309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.2.90.129:37147)
0620 22:54:21.520350 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.520351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.520358 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:54:21.520403 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.520404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.520419 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.520479 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:54:21.520582 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.521054 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.521714 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.521891 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.521932 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.522041 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:54:21.522130 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:54:21.522183 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.522198 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.522212 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:54:21.523743 16536 raft_consensus.cc:1261] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Refusing update from remote peer 56bad906eb00437b85726b3409c40eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:21.523751 16402 raft_consensus.cc:1261] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Refusing update from remote peer 56bad906eb00437b85726b3409c40eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:21.524025 16604 consensus_queue.cc:1035] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:21.524147 16607 consensus_queue.cc:1035] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:21.526705 16612 mvcc.cc:205] Tried to move back new op lower bound from 7299055867999092736 to 7299055867932950528. Current Snapshot: MvccSnapshot[applied={T|T < 7299055867999092736}]
I20260620 22:54:21.527251 16611 mvcc.cc:205] Tried to move back new op lower bound from 7299055867999092736 to 7299055867932950528. Current Snapshot: MvccSnapshot[applied={T|T < 7299055867999092736}]
I20260620 22:54:21.528277 16123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:21.528368 16123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:21.529074 16123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Sending DeleteTablet for 3 replicas of tablet 949af1efc6404801bd4d4d2cb43ad7fa
I20260620 22:54:21.529394 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16185
I20260620 22:54:21.529398 16248 tablet_service.cc:1514] Processing DeleteTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:21 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40228
I20260620 22:54:21.529407 16516 tablet_service.cc:1514] Processing DeleteTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:21 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:44730
I20260620 22:54:21.530061 16382 tablet_service.cc:1514] Processing DeleteTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:21 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:60010
I20260620 22:54:21.530251 16620 tablet_replica.cc:330] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: stopping tablet replica
I20260620 22:54:21.530325 16620 raft_consensus.cc:2229] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:21.530391 16620 raft_consensus.cc:2258] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:21.530488 16619 tablet_replica.cc:330] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: stopping tablet replica
I20260620 22:54:21.530570 16619 raft_consensus.cc:2229] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:21.530651 16619 raft_consensus.cc:2258] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:21.530746 16620 ts_tablet_manager.cc:1905] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:21.531069 16619 ts_tablet_manager.cc:1905] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:21.532147 16620 ts_tablet_manager.cc:1918] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:21.532218 16620 log.cc:1196] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/949af1efc6404801bd4d4d2cb43ad7fa
I20260620 22:54:21.532490 16620 ts_tablet_manager.cc:1939] T 949af1efc6404801bd4d4d2cb43ad7fa P d2cc2396ef084bba93b850f18a6314ba: Deleting consensus metadata
I20260620 22:54:21.532536 16619 ts_tablet_manager.cc:1918] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:21.532635 16619 log.cc:1196] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/949af1efc6404801bd4d4d2cb43ad7fa
I20260620 22:54:21.532898 16619 ts_tablet_manager.cc:1939] T 949af1efc6404801bd4d4d2cb43ad7fa P d619ce7db73e407988ee8112e2543ed4: Deleting consensus metadata
I20260620 22:54:21.532895 16109 catalog_manager.cc:4841] TS d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007): tablet 949af1efc6404801bd4d4d2cb43ad7fa (table test-table [id=8df589a315fa428ab1afcb92e1a47952]) successfully deleted
I20260620 22:54:21.533223 16110 catalog_manager.cc:4841] TS d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853): tablet 949af1efc6404801bd4d4d2cb43ad7fa (table test-table [id=8df589a315fa428ab1afcb92e1a47952]) successfully deleted
W20260620 22:54:21.535089 16107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:37147 (error 108)
I20260620 22:54:21.535480 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.535379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37147 (local address 127.0.0.1:40248)
0620 22:54:21.535420 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.535448 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:54:21.535557 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16319
W20260620 22:54:21.535576 16107 catalog_manager.cc:4584] TS 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 949af1efc6404801bd4d4d2cb43ad7fa: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37147: connect: Connection refused (error 111)
I20260620 22:54:21.539484 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16453
I20260620 22:54:21.543485 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16091
I20260620 22:54:21.547858 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44667
--webserver_interface=127.2.90.190
--webserver_port=39769
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:21.625062 16622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:21.625255 16622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:21.625275 16622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:21.626390 16622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:21.626431 16622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:21.626444 16622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:21.626467 16622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:21.626492 16622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:21.627933 16622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:21.628149 16622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:21.628343 16622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:21.636125 16629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:21.636125 16631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:21.636132 16628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.636371 16622 server_base.cc:1034] running on GCE node
I20260620 22:54:21.636634 16622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:21.636947 16622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:21.638078 16622 hybrid_clock.cc:648] HybridClock initialized: now 1781996061638069 us; error 25 us; skew 500 ppm
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:21.640357 16622 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:21.640769 16622 webserver.cc:460] Webserver started at http://127.2.90.190:39769/ using document root <none> and password file <none>
I20260620 22:54:21.640955 16622 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:21.641004 16622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:21.642221 16622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:21.642778 16638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.642921 16622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:21.643002 16622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "cebed98135a24b01b5051dc528b8d1e7"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7926bc036b95582b358d032c7c3e1416"
server_key_iv: "c3412a50a3fe11aa0d9e2b3cfefb4959"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.643376 16622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:21.654708 16622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:21.656790 16622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:21.656922 16622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:21.660610 16622 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:44667
I20260620 22:54:21.660643 16690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:44667 every 8 connection(s)
I20260620 22:54:21.661055 16622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:21.661747 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16622
I20260620 22:54:21.661991 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:37147
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:44667
--webserver_port=46409
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:21.662655 16691 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:54:21.663290 16691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap starting.
I20260620 22:54:21.671133 16691 log.cc:824] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:21.673137 16691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:21.673386 16691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap complete.
I20260620 22:54:21.675508 16691 raft_consensus.cc:348] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:21.675729 16691 raft_consensus.cc:729] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebed98135a24b01b5051dc528b8d1e7, State: Initialized, Role: FOLLOWER
I20260620 22:54:21.675874 16691 consensus_queue.cc:260] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:21.675931 16691 raft_consensus.cc:388] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:21.675989 16691 raft_consensus.cc:482] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:21.676035 16691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:54:21.677027 16691 raft_consensus.cc:504] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:21.677119 16691 leader_election.cc:302] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cebed98135a24b01b5051dc528b8d1e7; no voters:
I20260620 22:54:21.677275 16691 leader_election.cc:288] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:54:21.677320 16695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:54:21.677457 16695 raft_consensus.cc:686] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 LEADER]: Becoming Leader. State: Replica: cebed98135a24b01b5051dc528b8d1e7, State: Running, Role: LEADER
I20260620 22:54:21.677525 16691 sys_catalog.cc:564] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:21.677582 16695 consensus_queue.cc:237] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:21.677857 16696 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cebed98135a24b01b5051dc528b8d1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } } }
I20260620 22:54:21.677858 16695 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cebed98135a24b01b5051dc528b8d1e7. Latest consensus state: current_term: 2 leader_uuid: "cebed98135a24b01b5051dc528b8d1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } } }
I20260620 22:54:21.678016 16696 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:21.678048 16695 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:21.678251 16702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:21.679034 16702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:21.679136 16702 catalog_manager.cc:1259] Loaded cluster ID: 05aeb5716b634dca8e6fbd0a46c1bbf4
I20260620 22:54:21.679222 16702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:54:21.679702 16702 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 22:54:21.684326 16702 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 22:54:21.686586 16702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:21.686777 16702 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Loaded TSK: 0
I20260620 22:54:21.687000 16702 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:54:21.742125 16693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:21.742339 16693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:21.742372 16693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:21.743507 16693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:21.743547 16693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:21.743582 16693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:21.745038 16693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=46409
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:21.745267 16693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:21.745468 16693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:21.750219 16719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:21.750231 16721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.750329 16693 server_base.cc:1034] running on GCE node
W20260620 22:54:21.750242 16718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.750670 16693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:21.750950 16693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:21.752136 16693 hybrid_clock.cc:648] HybridClock initialized: now 1781996061752116 us; error 33 us; skew 500 ppm
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:21.755089 16693 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:21.755637 16693 webserver.cc:460] Webserver started at http://127.2.90.129:46409/ using document root <none> and password file <none>
I20260620 22:54:21.755893 16693 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:21.756004 16693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:21.757732 16693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:21.758468 16728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.758628 16693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:21.758723 16693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "56bad906eb00437b85726b3409c40eba"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7c1aa85de3cd8444637497a6c583f4fc"
server_key_iv: "e84050f04326753c2b6f61a7491e3e93"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.759159 16693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:21.777592 16693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:21.779975 16693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:21.780135 16693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:21.780406 16693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:21.780938 16735 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:54:21.782331 16693 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:54:21.782394 16693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:54:21.782430 16693 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:54:21.783279 16693 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:54:21.783325 16693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:21.783371 16735 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Bootstrap starting.
I20260620 22:54:21.790342 16693 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37147
I20260620 22:54:21.790402 16842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37147 every 8 connection(s)
I20260620 22:54:21.790802 16693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:54:21.793300 16735 log.cc:824] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:21.795816 16735 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:21.796072 16735 tablet_bootstrap.cc:492] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Bootstrap complete.
I20260620 22:54:21.796546 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16693
I20260620 22:54:21.796829 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:45007
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:44667
--webserver_port=40843
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:21.797087 16735 ts_tablet_manager.cc:1397] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20260620 22:54:21.798688 16735 raft_consensus.cc:348] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.799037 16735 raft_consensus.cc:729] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bad906eb00437b85726b3409c40eba, State: Initialized, Role: FOLLOWER
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:21.799190 16735 consensus_queue.cc:260] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:21.800150 16735 ts_tablet_manager.cc:1428] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 22:54:21.808993 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.791559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:33153 (local address 127.2.90.190:44667)
0620 22:54:21.792823 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.792833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.792874 (+ 41us) server_negotiation.cc:407] Connection header received
0620 22:54:21.793020 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.793027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.793087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.793238 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:21.794046 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.795406 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.796972 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.797595 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.797699 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.800372 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:21.800376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:21.800377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:21.800389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:21.807010 (+ 6621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.807520 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.807524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.807524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.807640 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.807916 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.807917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.807918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.808213 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:21.808291 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.808605 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.808793 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":66,"threads_started":1}
I20260620 22:54:21.809051 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.791481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:33153)
0620 22:54:21.791671 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.791715 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.792293 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.793302 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.793306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.793657 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:21.793920 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.793927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.795551 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.795553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.796872 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.796875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.797815 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.797816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.797958 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.798538 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:21.798556 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:21.800252 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:21.807122 (+ 6870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.807131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.807134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.807430 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.807706 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.807707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.807707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.807855 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.808280 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:21.808286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:21.808440 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.808550 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.808747 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260620 22:54:21.809742 16843 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44667
I20260620 22:54:21.809867 16843 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:21.810065 16843 heartbeater.cc:511] Master 127.2.90.190:44667 requested a full tablet report, sending...
I20260620 22:54:21.810842 16655 ts_manager.cc:194] Registered new tserver with Master: 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147)
I20260620 22:54:21.811988 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:33153
I20260620 22:54:21.812947 16843 heartbeater.cc:503] Master 127.2.90.190:44667 was elected leader, sending a full tablet report...
I20260620 22:54:21.817185 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.811667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.2.90.129:37147)
0620 22:54:21.811814 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.811818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.813098 (+ 1280us) server_negotiation.cc:407] Connection header received
0620 22:54:21.813299 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.813301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.813316 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.813375 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:21.813592 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.814651 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.816138 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.816804 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.816863 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.816911 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.817083 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.817116 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260620 22:54:21.817189 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.811430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37147 (local address 127.0.0.1:40260)
0620 22:54:21.812290 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.812320 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.812385 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.813452 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.813454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.813456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:21.813514 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.813515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.814727 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.814728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.816051 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.816054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.816888 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.816889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.816989 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.817003 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.817066 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.817115 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":92,"threads_started":1}
I20260620 22:54:21.817786 16772 tablet_service.cc:1514] Processing DeleteTablet for tablet 949af1efc6404801bd4d4d2cb43ad7fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:21 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40260
I20260620 22:54:21.818408 16852 tablet_replica.cc:330] stopping tablet replica
I20260620 22:54:21.818512 16852 raft_consensus.cc:2229] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:21.818573 16852 raft_consensus.cc:2258] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:21.818876 16852 ts_tablet_manager.cc:1905] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:21.821139 16852 ts_tablet_manager.cc:1918] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:21.821231 16852 log.cc:1196] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/949af1efc6404801bd4d4d2cb43ad7fa
I20260620 22:54:21.821445 16852 ts_tablet_manager.cc:1939] T 949af1efc6404801bd4d4d2cb43ad7fa P 56bad906eb00437b85726b3409c40eba: Deleting consensus metadata
I20260620 22:54:21.821883 16639 catalog_manager.cc:4841] TS 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147): tablet 949af1efc6404801bd4d4d2cb43ad7fa (table test-table [id=8df589a315fa428ab1afcb92e1a47952]) successfully deleted
W20260620 22:54:21.879916 16848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:21.880102 16848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:21.880123 16848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:21.881229 16848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:21.881273 16848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:21.881299 16848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:54:21.882695 16848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:45007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=40843
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:21.882877 16848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:21.883045 16848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:21.887530 16860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:21.887544 16858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.887759 16848 server_base.cc:1034] running on GCE node
W20260620 22:54:21.887544 16857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:21.887959 16848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:21.888165 16848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:21.889292 16848 hybrid_clock.cc:648] HybridClock initialized: now 1781996061889275 us; error 26 us; skew 500 ppm
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:21.891440 16848 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:54:21.891832 16848 webserver.cc:460] Webserver started at http://127.2.90.130:40843/ using document root <none> and password file <none>
I20260620 22:54:21.892033 16848 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:21.892094 16848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:21.893402 16848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:21.893949 16867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.894095 16848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:54:21.894174 16848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "d2cc2396ef084bba93b850f18a6314ba"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "dbf5fdd8c279134ee96f713da5f0e1ab"
server_key_iv: "605e02ac87960612ab75ad8cafa6b10f"
server_key_version: "encryptionkey@0"
I20260620 22:54:21.894492 16848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:21.911631 16848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:21.913480 16848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:21.913612 16848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:21.913825 16848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:21.914099 16848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:21.914132 16848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.914160 16848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:21.914178 16848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:21.919275 16848 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45007
I20260620 22:54:21.919338 16980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45007 every 8 connection(s)
I20260620 22:54:21.919662 16848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:54:21.922398 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16848
I20260620 22:54:21.922667 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:42853
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:44667
--webserver_port=38869
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:54:21 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996061, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:21.929138 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.920368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:54309 (local address 127.2.90.190:44667)
0620 22:54:21.920420 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:21.920423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:21.920607 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:54:21.921135 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:21.921136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:21.921150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:21.921201 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:21.922074 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.922732 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.924093 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.924396 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.924479 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.926653 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:21.926654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:21.926654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:21.926661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:21.927835 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.928264 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.928265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.928265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.928306 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:21.928552 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:21.928552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:21.928552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:21.928640 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:21.928705 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:21.929009 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 22:54:21.929075 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:54:21.929373 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:21.920341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:54309)
0620 22:54:21.920500 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:21.920535 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:21.921044 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:21.921267 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:21.921272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:21.921801 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:21.921984 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.921989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.922837 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.922838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.924011 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:21.924014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:21.924470 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:21.924471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:21.924563 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:21.925067 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:21.925078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:21.926567 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:21.927908 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.927915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.927917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.928198 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.928358 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:21.928359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:21.928359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:21.928497 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:21.928713 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:21.928718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:21.928878 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 22:54:21.928991 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 22:54:21.929149 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260620 22:54:21.929903 16981 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44667
I20260620 22:54:21.929993 16981 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:21.930159 16981 heartbeater.cc:511] Master 127.2.90.190:44667 requested a full tablet report, sending...
I20260620 22:54:21.930521 16655 ts_manager.cc:194] Registered new tserver with Master: d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007)
I20260620 22:54:21.930994 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:54309
W20260620 22:54:22.001825 16984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:22.002030 16984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:22.002053 16984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:22.003161 16984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:22.003201 16984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:22.003237 16984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:54:22.004726 16984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:42853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=38869
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:22.004949 16984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:22.005146 16984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:22.009622 16990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.009689 16992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.009711 16989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.009820 16984 server_base.cc:1034] running on GCE node
I20260620 22:54:22.010037 16984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:22.010268 16984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:22.011397 16984 hybrid_clock.cc:648] HybridClock initialized: now 1781996062011382 us; error 29 us; skew 500 ppm
Jun 20 22:54:22 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:22.013746 16984 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:54:22.014163 16984 webserver.cc:460] Webserver started at http://127.2.90.131:38869/ using document root <none> and password file <none>
I20260620 22:54:22.014353 16984 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:22.014413 16984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:22.015609 16984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:54:22.016091 16999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.016235 16984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.016312 16984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d619ce7db73e407988ee8112e2543ed4"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "096773c66d7d5a584e916e99708c8b3f"
server_key_iv: "786cc823b7892725fe66701de9c3a887"
server_key_version: "encryptionkey@0"
I20260620 22:54:22.016618 16984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:22.031716 16984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:22.033739 16984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:22.033864 16984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:22.034065 16984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:22.034330 16984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:22.034368 16984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.034394 16984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:22.034418 16984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.039619 16984 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42853
I20260620 22:54:22.039655 17112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42853 every 8 connection(s)
I20260620 22:54:22.039974 16984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
Jun 20 22:54:22 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:22.047719 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 16984
I20260620 22:54:22.048401 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.040640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:55151 (local address 127.2.90.190:44667)
0620 22:54:22.040733 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.040736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.040894 (+ 158us) server_negotiation.cc:407] Connection header received
0620 22:54:22.041387 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.041389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.041400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.041452 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:22.042340 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.042822 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.043787 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.043978 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.044031 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.046028 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:22.046029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:22.046029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:22.046036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:22.047168 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:22.047500 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:22.047501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:22.047501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:22.047543 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:22.047736 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:22.047737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:22.047737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:22.047870 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:22.047941 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.048259 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.048339 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 22:54:22.048547 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.040597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:55151)
0620 22:54:22.040777 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.040821 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:22.041296 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:22.041523 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:22.041527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:22.042061 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:22.042249 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.042254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.042900 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.042901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.043691 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.043693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.044044 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.044045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.044161 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.044570 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:22.044577 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:22.045955 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:22.047237 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:22.047243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:22.047245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:22.047445 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:22.047593 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:22.047594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:22.047594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:22.047685 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:22.047938 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:22.047944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:22.048129 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 22:54:22.048235 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 22:54:22.048352 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260620 22:54:22.049104 17113 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44667
I20260620 22:54:22.049193 17113 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:22.049355 17113 heartbeater.cc:511] Master 127.2.90.190:44667 requested a full tablet report, sending...
I20260620 22:54:22.049695 16655 ts_manager.cc:194] Registered new tserver with Master: d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853)
I20260620 22:54:22.050150 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:55151
I20260620 22:54:22.053998 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.048067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.2.90.190:44667)
0620 22:54:22.048172 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.048173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.048472 (+ 299us) server_negotiation.cc:407] Connection header received
0620 22:54:22.048492 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.048492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.048503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.048557 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:22.049198 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.049729 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.050668 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.050874 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.050926 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.051945 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:22.051946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:22.051947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:22.051953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:22.053050 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:22.053329 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:22.053330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:22.053330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:22.053384 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:22.053543 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:22.053544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:22.053544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:22.053645 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:22.053796 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.053830 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.053901 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260620 22:54:22.054541 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:54:22.057067 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.054841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.2.90.190:44667)
0620 22:54:22.054876 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.054877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.054969 (+ 92us) server_negotiation.cc:407] Connection header received
0620 22:54:22.055033 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.055034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.055070 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.055125 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:54:22.055220 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.055672 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.056369 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.056570 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.056621 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.056791 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:54:22.056879 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:54:22.056943 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.056989 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.057008 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:54:22.057315 16655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:54:22.058794 16655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:54:22.060595 16772 tablet_service.cc:1467] Processing CreateTablet for tablet 4f51ce18e3f248f08f737fe5b02fe95c (DEFAULT_TABLE table=test-table [id=f157fac99d0645a2b0c9c61fcea12b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:22.060783 16772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f51ce18e3f248f08f737fe5b02fe95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:22.062167 16735 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Bootstrap starting.
I20260620 22:54:22.062575 16735 tablet_bootstrap.cc:654] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:22.063174 16735 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: No bootstrap required, opened a new log
I20260620 22:54:22.063239 16735 ts_tablet_manager.cc:1397] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 22:54:22.063383 16735 raft_consensus.cc:348] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.063443 16735 raft_consensus.cc:374] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:22.063486 16735 raft_consensus.cc:729] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bad906eb00437b85726b3409c40eba, State: Initialized, Role: FOLLOWER
I20260620 22:54:22.063531 16735 consensus_queue.cc:260] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.063619 16735 ts_tablet_manager.cc:1428] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.064217 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.060344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42853 (local address 127.0.0.1:44744)
0620 22:54:22.060499 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.060507 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:22.060541 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:22.060962 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:22.060964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:22.060966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:22.061020 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.061022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.062331 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.062332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.063196 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.063198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.064013 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.064014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.064076 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.064080 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:54:22.064107 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 22:54:22.064144 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":56,"thread_start_us":88,"threads_started":1}
I20260620 22:54:22.064522 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.060400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.2.90.131:42853)
0620 22:54:22.060808 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.060812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.060820 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:54:22.060849 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.060851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.060866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.060948 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:22.061080 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.062219 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.063282 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.063922 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.064317 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.064370 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.064395 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.064446 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":79,"threads_started":1}
I20260620 22:54:22.064600 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.060344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45007 (local address 127.0.0.1:60012)
0620 22:54:22.060489 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.060499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:22.060539 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:22.060971 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:22.060972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:22.060973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:22.061009 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.061010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.062322 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.062323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.063430 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.063433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.064396 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.064397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.064452 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.064455 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:54:22.064507 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 22:54:22.064546 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":29}
I20260620 22:54:22.064659 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.060620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.2.90.130:45007)
0620 22:54:22.060797 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.060802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.060812 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:54:22.060850 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.060851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.060866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.061075 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:22.061146 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.062250 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.063527 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.064237 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.064292 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.064351 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.064535 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.064573 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260620 22:54:22.065157 17047 tablet_service.cc:1467] Processing CreateTablet for tablet 4f51ce18e3f248f08f737fe5b02fe95c (DEFAULT_TABLE table=test-table [id=f157fac99d0645a2b0c9c61fcea12b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:22.065277 16915 tablet_service.cc:1467] Processing CreateTablet for tablet 4f51ce18e3f248f08f737fe5b02fe95c (DEFAULT_TABLE table=test-table [id=f157fac99d0645a2b0c9c61fcea12b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:22.065533 16915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f51ce18e3f248f08f737fe5b02fe95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:22.065542 17047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f51ce18e3f248f08f737fe5b02fe95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:22.067983 17123 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Bootstrap starting.
I20260620 22:54:22.068215 17124 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Bootstrap starting.
I20260620 22:54:22.068581 17123 tablet_bootstrap.cc:654] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:22.068825 17124 tablet_bootstrap.cc:654] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:22.069152 17123 log.cc:824] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:22.069314 17124 log.cc:824] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:22.069873 17123 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: No bootstrap required, opened a new log
I20260620 22:54:22.069936 17123 ts_tablet_manager.cc:1397] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:54:22.069962 17124 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: No bootstrap required, opened a new log
I20260620 22:54:22.070026 17124 ts_tablet_manager.cc:1397] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:54:22.071254 17123 raft_consensus.cc:348] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.071295 17124 raft_consensus.cc:348] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.071393 17123 raft_consensus.cc:374] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:22.071411 17124 raft_consensus.cc:374] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:22.071426 17123 raft_consensus.cc:729] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cc2396ef084bba93b850f18a6314ba, State: Initialized, Role: FOLLOWER
I20260620 22:54:22.071442 17124 raft_consensus.cc:729] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d619ce7db73e407988ee8112e2543ed4, State: Initialized, Role: FOLLOWER
I20260620 22:54:22.071552 17124 consensus_queue.cc:260] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.071554 17123 consensus_queue.cc:260] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.071794 17123 ts_tablet_manager.cc:1428] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:54:22.071805 17124 ts_tablet_manager.cc:1428] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:54:22.071842 16981 heartbeater.cc:503] Master 127.2.90.190:44667 was elected leader, sending a full tablet report...
I20260620 22:54:22.071909 17113 heartbeater.cc:503] Master 127.2.90.190:44667 was elected leader, sending a full tablet report...
I20260620 22:54:22.084787 17127 raft_consensus.cc:482] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:54:22.084870 17127 raft_consensus.cc:504] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.085130 17127 leader_election.cc:288] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007), 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147)
I20260620 22:54:22.088721 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.085338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:35677 (local address 127.2.90.129:37147)
0620 22:54:22.085377 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.085379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.085487 (+ 108us) server_negotiation.cc:407] Connection header received
0620 22:54:22.085520 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.085521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.085531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.085580 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:22.085688 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.086378 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.087797 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.088356 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.088426 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.088432 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.088590 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.088630 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:54:22.088737 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.085263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45007 (local address 127.2.90.131:37417)
0620 22:54:22.085314 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.085325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:22.085362 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:22.085525 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:22.085526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:22.085528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:22.085589 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.085591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.086447 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.086447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.087712 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.087715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.088444 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.088445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.088540 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.088543 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:54:22.088585 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:54:22.088617 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":64}
I20260620 22:54:22.088763 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.085320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37417 (local address 127.2.90.130:45007)
0620 22:54:22.085357 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.085359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.085367 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:54:22.085453 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.085454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.085461 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.085512 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:22.085649 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.086383 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.087816 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.088371 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.088424 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.088432 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.088645 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.088710 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:54:22.088737 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.085274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37147 (local address 127.2.90.131:35677)
0620 22:54:22.085423 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.085432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:22.085480 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:22.085586 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:22.085586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:22.085588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:22.085635 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.085636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.086444 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.086444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.087713 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:22.087715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.088430 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.088431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:22.088516 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.088521 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:54:22.088568 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 22:54:22.088611 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":53,"thread_start_us":86,"threads_started":1}
I20260620 22:54:22.089035 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f51ce18e3f248f08f737fe5b02fe95c" candidate_uuid: "d619ce7db73e407988ee8112e2543ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cc2396ef084bba93b850f18a6314ba" is_pre_election: true
I20260620 22:54:22.089152 16935 raft_consensus.cc:2454] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d619ce7db73e407988ee8112e2543ed4 in term 0.
I20260620 22:54:22.089345 17002 leader_election.cc:302] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2cc2396ef084bba93b850f18a6314ba, d619ce7db73e407988ee8112e2543ed4; no voters:
I20260620 22:54:22.089445 17127 raft_consensus.cc:2781] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:54:22.089496 17127 raft_consensus.cc:482] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:54:22.089514 17127 raft_consensus.cc:3037] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:22.089442 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f51ce18e3f248f08f737fe5b02fe95c" candidate_uuid: "d619ce7db73e407988ee8112e2543ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bad906eb00437b85726b3409c40eba" is_pre_election: true
I20260620 22:54:22.089593 16793 raft_consensus.cc:2454] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d619ce7db73e407988ee8112e2543ed4 in term 0.
I20260620 22:54:22.090189 17127 raft_consensus.cc:504] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.090296 17127 leader_election.cc:288] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [CANDIDATE]: Term 1 election: Requested vote from peers d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007), 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147)
I20260620 22:54:22.090415 16935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f51ce18e3f248f08f737fe5b02fe95c" candidate_uuid: "d619ce7db73e407988ee8112e2543ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cc2396ef084bba93b850f18a6314ba"
I20260620 22:54:22.090428 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f51ce18e3f248f08f737fe5b02fe95c" candidate_uuid: "d619ce7db73e407988ee8112e2543ed4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bad906eb00437b85726b3409c40eba"
I20260620 22:54:22.090492 16935 raft_consensus.cc:3037] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:22.090494 16793 raft_consensus.cc:3037] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:22.091122 16793 raft_consensus.cc:2454] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d619ce7db73e407988ee8112e2543ed4 in term 1.
I20260620 22:54:22.091151 16935 raft_consensus.cc:2454] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d619ce7db73e407988ee8112e2543ed4 in term 1.
I20260620 22:54:22.091279 17000 leader_election.cc:302] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56bad906eb00437b85726b3409c40eba, d619ce7db73e407988ee8112e2543ed4; no voters:
I20260620 22:54:22.091377 17127 raft_consensus.cc:2781] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:22.091517 17127 raft_consensus.cc:686] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 1 LEADER]: Becoming Leader. State: Replica: d619ce7db73e407988ee8112e2543ed4, State: Running, Role: LEADER
I20260620 22:54:22.091609 17127 consensus_queue.cc:237] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.092340 16653 catalog_manager.cc:5495] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d619ce7db73e407988ee8112e2543ed4 (127.2.90.131). New cstate: current_term: 1 leader_uuid: "d619ce7db73e407988ee8112e2543ed4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } health_report { overall_health: UNKNOWN } } }
I20260620 22:54:22.099399 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.097384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.2.90.131:42853)
0620 22:54:22.097419 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:22.097421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:22.097456 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:54:22.097539 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:22.097540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:22.097554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:22.097604 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:54:22.097692 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.098115 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.098780 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:22.099021 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:22.099066 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:22.099195 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:54:22.099275 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:54:22.099322 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:22.099340 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 22:54:22.099354 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:54:22.100903 16793 raft_consensus.cc:1261] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Refusing update from remote peer d619ce7db73e407988ee8112e2543ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:22.100903 16935 raft_consensus.cc:1261] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Refusing update from remote peer d619ce7db73e407988ee8112e2543ed4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:22.101188 17130 consensus_queue.cc:1035] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:22.101305 17127 consensus_queue.cc:1035] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:22.103255 17133 mvcc.cc:205] Tried to move back new op lower bound from 7299055870363185152 to 7299055870327877632. Current Snapshot: MvccSnapshot[applied={T|T < 7299055870363185152}]
I20260620 22:54:22.104221 17134 mvcc.cc:205] Tried to move back new op lower bound from 7299055870363185152 to 7299055870327877632. Current Snapshot: MvccSnapshot[applied={T|T < 7299055870363185152}]
I20260620 22:54:22.105417 16655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:22.105506 16655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:22.106154 16655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Sending DeleteTablet for 3 replicas of tablet 4f51ce18e3f248f08f737fe5b02fe95c
I20260620 22:54:22.106483 17047 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f51ce18e3f248f08f737fe5b02fe95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:22 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:44744
I20260620 22:54:22.106496 16915 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f51ce18e3f248f08f737fe5b02fe95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:22 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:60012
I20260620 22:54:22.106532 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16693
I20260620 22:54:22.106655 16772 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f51ce18e3f248f08f737fe5b02fe95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:22 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40260
I20260620 22:54:22.107530 17143 tablet_replica.cc:330] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: stopping tablet replica
I20260620 22:54:22.107600 17143 raft_consensus.cc:2229] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:54:22.107659 17143 raft_consensus.cc:2258] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:22.108011 17143 ts_tablet_manager.cc:1905] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:22.109385 17143 ts_tablet_manager.cc:1918] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:22.109455 17143 log.cc:1196] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/4f51ce18e3f248f08f737fe5b02fe95c
I20260620 22:54:22.109753 17143 ts_tablet_manager.cc:1939] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Deleting consensus metadata
I20260620 22:54:22.110128 16642 catalog_manager.cc:4841] TS d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853): tablet 4f51ce18e3f248f08f737fe5b02fe95c (table test-table [id=f157fac99d0645a2b0c9c61fcea12b6d]) successfully deleted
W20260620 22:54:22.112005 16639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:37147 (error 108)
I20260620 22:54:22.112206 17144 tablet_replica.cc:330] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: stopping tablet replica
I20260620 22:54:22.112331 17144 raft_consensus.cc:2229] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:22.112438 17144 raft_consensus.cc:2258] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:22.112528 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.112416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37147 (local address 127.0.0.1:40274)
0620 22:54:22.112460 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.112496 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:54:22.112635 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16848
W20260620 22:54:22.112731 16639 catalog_manager.cc:4584] TS 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f51ce18e3f248f08f737fe5b02fe95c: Network error: Client connection negotiation failed: client connection to 127.2.90.129:37147: connect: Connection refused (error 111)
I20260620 22:54:22.118913 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16984
I20260620 22:54:22.118966 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.118844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45007 (local address 127.0.0.1:60024)
0620 22:54:22.118891 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.118924 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:54:22.123152 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 16622
I20260620 22:54:22.127482 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44667
--webserver_interface=127.2.90.190
--webserver_port=39769
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:22.212598 17146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:22.212831 17146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:22.212854 17146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:22.213999 17146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:22.214035 17146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:22.214048 17146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:22.214061 17146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:22.214085 17146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:22.215513 17146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=39769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:22.215736 17146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:22.215941 17146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:22.220332 17154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.220448 17152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.220368 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.220420 17146 server_base.cc:1034] running on GCE node
I20260620 22:54:22.220803 17146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:22.221071 17146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:22.222208 17146 hybrid_clock.cc:648] HybridClock initialized: now 1781996062222181 us; error 39 us; skew 500 ppm
Jun 20 22:54:22 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:22.224455 17146 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:22.224884 17146 webserver.cc:460] Webserver started at http://127.2.90.190:39769/ using document root <none> and password file <none>
I20260620 22:54:22.225071 17146 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:22.225118 17146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:22.226342 17146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:22.226944 17161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.227104 17146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:22.227190 17146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "cebed98135a24b01b5051dc528b8d1e7"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7926bc036b95582b358d032c7c3e1416"
server_key_iv: "c3412a50a3fe11aa0d9e2b3cfefb4959"
server_key_version: "encryptionkey@0"
I20260620 22:54:22.227528 17146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:22.236532 17146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:22.238509 17146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:22.238636 17146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:22.242233 17146 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:44667
I20260620 22:54:22.242264 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:44667 every 8 connection(s)
I20260620 22:54:22.242594 17146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:22.244027 17214 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:54:22.244642 17214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap starting.
I20260620 22:54:22.251820 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17146
I20260620 22:54:22.252148 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:37147
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:44667
--webserver_port=46409
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:22.252879 17214 log.cc:824] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:22.255219 17214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:22.255549 17214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7: Bootstrap complete.
I20260620 22:54:22.257652 17214 raft_consensus.cc:348] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:22.257881 17214 raft_consensus.cc:729] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebed98135a24b01b5051dc528b8d1e7, State: Initialized, Role: FOLLOWER
I20260620 22:54:22.258029 17214 consensus_queue.cc:260] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:22.258105 17214 raft_consensus.cc:388] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:22.258154 17214 raft_consensus.cc:482] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:22.258200 17214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:54:22.259147 17214 raft_consensus.cc:504] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:22.259238 17214 leader_election.cc:302] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cebed98135a24b01b5051dc528b8d1e7; no voters:
I20260620 22:54:22.259420 17214 leader_election.cc:288] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:54:22.259490 17218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:54:22.259675 17214 sys_catalog.cc:564] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:22.259693 17218 raft_consensus.cc:686] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [term 3 LEADER]: Becoming Leader. State: Replica: cebed98135a24b01b5051dc528b8d1e7, State: Running, Role: LEADER
I20260620 22:54:22.259794 17218 consensus_queue.cc:237] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } }
I20260620 22:54:22.260149 17218 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cebed98135a24b01b5051dc528b8d1e7. Latest consensus state: current_term: 3 leader_uuid: "cebed98135a24b01b5051dc528b8d1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } } }
I20260620 22:54:22.260244 17218 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:22.260792 17219 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cebed98135a24b01b5051dc528b8d1e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebed98135a24b01b5051dc528b8d1e7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44667 } } }
I20260620 22:54:22.260864 17219 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebed98135a24b01b5051dc528b8d1e7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:22.260865 17227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:22.261785 17227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:22.261929 17227 catalog_manager.cc:1259] Loaded cluster ID: 05aeb5716b634dca8e6fbd0a46c1bbf4
I20260620 22:54:22.262125 17227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 22:54:22.262394 17227 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f51bbaec9ad google::LogMessage::Fail() at ??:0
@ 0x7f51bbaef88a google::LogMessage::SendToLog() at ??:0
@ 0x7f51bbaec4de google::LogMessage::Flush() at ??:0
@ 0x7f51bbaeda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f51c3a0d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f51c3a168b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f51bd9ba0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f51bd9af954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f51be8136db start_thread at ??:0
@ 0x7f51ba5ea61f clone at ??:0
W20260620 22:54:22.361512 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:22.361789 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:22.361828 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:22.363541 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:22.363603 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:22.363652 17217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:22.365805 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=46409
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:22.366060 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:22.366318 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:22.371317 17250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.371312 17251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.371483 17253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.371491 17217 server_base.cc:1034] running on GCE node
I20260620 22:54:22.371729 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:22.371948 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:22.373078 17217 hybrid_clock.cc:648] HybridClock initialized: now 1781996062373067 us; error 32 us; skew 500 ppm
Jun 20 22:54:22 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:22.375455 17217 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:22.375878 17217 webserver.cc:460] Webserver started at http://127.2.90.129:46409/ using document root <none> and password file <none>
I20260620 22:54:22.376086 17217 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:22.376134 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:22.377488 17217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:22.378120 17260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.378266 17217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:22.378343 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "56bad906eb00437b85726b3409c40eba"
format_stamp: "Formatted at 2026-06-20 22:54:20 on dist-test-slave-4mtw"
server_key: "7c1aa85de3cd8444637497a6c583f4fc"
server_key_iv: "e84050f04326753c2b6f61a7491e3e93"
server_key_version: "encryptionkey@0"
I20260620 22:54:22.378644 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:22.403620 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:22.405736 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:22.405874 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:22.406088 17217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:22.406500 17267 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:54:22.408393 17217 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:54:22.408437 17217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:54:22.408470 17217 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:54:22.409121 17217 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:54:22.409153 17217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:22.409235 17267 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Bootstrap starting.
I20260620 22:54:22.415560 17217 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:37147
I20260620 22:54:22.415686 17374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:37147 every 8 connection(s)
I20260620 22:54:22.415968 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:54:22.417086 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17217
I20260620 22:54:22.417335 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:45007
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:44667
--webserver_port=40843
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:22.419638 17267 log.cc:824] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:22.422053 17267 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:22.422307 17267 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Bootstrap complete.
I20260620 22:54:22.423031 17267 ts_tablet_manager.cc:1397] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260620 22:54:22.424207 17267 raft_consensus.cc:348] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.424450 17267 raft_consensus.cc:729] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bad906eb00437b85726b3409c40eba, State: Initialized, Role: FOLLOWER
I20260620 22:54:22.424589 17267 consensus_queue.cc:260] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.424901 17267 ts_tablet_manager.cc:1428] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
W20260620 22:54:22.472016 17377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:54:22.416640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:42045)
0620 22:54:22.424753 (+ 8113us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.424799 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:22.425021 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:22.471805 (+ 46784us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7979,"thread_start_us":109,"threads_started":1}
W20260620 22:54:22.472291 17375 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:44667 (0 consecutive failures): Network error: Failed to ping master at 127.2.90.190:44667: Client connection negotiation failed: client connection to 127.2.90.190:44667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:54:22.472597 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.472495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:39267)
0620 22:54:22.472541 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.472577 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:54:22.472963 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.472853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:49677)
0620 22:54:22.472920 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.472945 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 22:54:22.473066 17375 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:54:22.504947 17378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:22.505124 17378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:22.505143 17378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:22.506215 17378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:22.506248 17378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:22.506274 17378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:54:22.507639 17378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:45007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=40843
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:22.507814 17378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:22.507982 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:22.512451 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.512460 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.512646 17378 server_base.cc:1034] running on GCE node
W20260620 22:54:22.512825 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.513018 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:22.513207 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:22.514367 17378 hybrid_clock.cc:648] HybridClock initialized: now 1781996062514355 us; error 24 us; skew 500 ppm
Jun 20 22:54:22 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:22.516571 17378 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:54:22.516983 17378 webserver.cc:460] Webserver started at http://127.2.90.130:40843/ using document root <none> and password file <none>
I20260620 22:54:22.517161 17378 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:22.517199 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:22.518352 17378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:54:22.518890 17395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.519037 17378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:54:22.519109 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "d2cc2396ef084bba93b850f18a6314ba"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "dbf5fdd8c279134ee96f713da5f0e1ab"
server_key_iv: "605e02ac87960612ab75ad8cafa6b10f"
server_key_version: "encryptionkey@0"
I20260620 22:54:22.519412 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:22.537436 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:22.539170 17378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:22.539299 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:22.539521 17378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:22.539914 17402 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:54:22.541332 17378 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:54:22.541374 17378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:54:22.541406 17378 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:54:22.542022 17378 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:54:22.542054 17378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:22.542100 17402 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Bootstrap starting.
I20260620 22:54:22.548532 17378 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:45007
I20260620 22:54:22.548620 17509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:45007 every 8 connection(s)
I20260620 22:54:22.548946 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:54:22.550040 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.549534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:52501)
0620 22:54:22.549711 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.549821 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
W20260620 22:54:22.550271 17510 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:44667 (0 consecutive failures): Network error: Failed to ping master at 127.2.90.190:44667: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
I20260620 22:54:22.550550 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.550451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:43373)
0620 22:54:22.550499 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.550528 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:54:22.550776 17402 log.cc:824] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:22.550830 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.550746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:33697)
0620 22:54:22.550789 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.550814 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:54:22.550936 17510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:54:22.552822 17402 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:22.553005 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17378
I20260620 22:54:22.553103 17402 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Bootstrap complete.
I20260620 22:54:22.553264 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:42853
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:44667
--webserver_port=38869
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:22.554189 17402 ts_tablet_manager.cc:1397] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260620 22:54:22.555826 17402 raft_consensus.cc:348] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.556134 17402 raft_consensus.cc:729] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cc2396ef084bba93b850f18a6314ba, State: Initialized, Role: FOLLOWER
I20260620 22:54:22.556262 17402 consensus_queue.cc:260] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:22.556489 17402 ts_tablet_manager.cc:1428] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260620 22:54:22.556725 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.556617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:53215)
0620 22:54:22.556650 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.556703 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 22:54:22.556970 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.556891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:45403)
0620 22:54:22.556932 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.556954 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 22:54:22.632516 17514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:22.632732 17514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:22.632763 17514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:22.633874 17514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:22.633911 17514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:22.633939 17514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:54:22.635316 17514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:42853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=38869
--tserver_master_addrs=127.2.90.190:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:22.635530 17514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:22.635730 17514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:22.640268 17523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.640352 17514 server_base.cc:1034] running on GCE node
W20260620 22:54:22.640302 17521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:22.640246 17520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:22.640657 17514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:22.640918 17514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:22.642051 17514 hybrid_clock.cc:648] HybridClock initialized: now 1781996062642044 us; error 28 us; skew 500 ppm
Jun 20 22:54:22 dist-test-slave-4mtw krb5kdc[16078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:22.644306 17514 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:54:22.644671 17514 webserver.cc:460] Webserver started at http://127.2.90.131:38869/ using document root <none> and password file <none>
I20260620 22:54:22.644889 17514 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:22.644932 17514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:22.646098 17514 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:54:22.646606 17530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.646751 17514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.646826 17514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d619ce7db73e407988ee8112e2543ed4"
format_stamp: "Formatted at 2026-06-20 22:54:21 on dist-test-slave-4mtw"
server_key: "096773c66d7d5a584e916e99708c8b3f"
server_key_iv: "786cc823b7892725fe66701de9c3a887"
server_key_version: "encryptionkey@0"
I20260620 22:54:22.647126 17514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:22.657135 17514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:22.658705 17514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:22.658809 17514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:22.658979 17514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:22.659253 17514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:22.659284 17514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.659304 17514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:22.659320 17514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:22.664716 17514 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42853
I20260620 22:54:22.664773 17643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42853 every 8 connection(s)
I20260620 22:54:22.665081 17514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:54:22.666049 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.665697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:49763)
0620 22:54:22.665836 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.665907 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
W20260620 22:54:22.666407 17644 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:44667 (0 consecutive failures): Network error: Failed to ping master at 127.2.90.190:44667: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
I20260620 22:54:22.666653 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.666562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:50489)
0620 22:54:22.666604 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.666634 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 22:54:22.666885 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:22.666813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:42637)
0620 22:54:22.666846 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:22.666868 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 22:54:22.666965 17644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:54:22.669294 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17514
I20260620 22:54:23.473574 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:23.473368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:33985)
0620 22:54:23.473494 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:23.473525 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260620 22:54:23.557494 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:23.557268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:42093)
0620 22:54:23.557411 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:23.557451 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260620 22:54:23.667490 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:23.667269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:53729)
0620 22:54:23.667406 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:23.667439 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260620 22:54:24.031380 17651 raft_consensus.cc:482] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:54:24.031519 17651 raft_consensus.cc:504] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:24.031795 17651 leader_election.cc:288] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853), d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007)
Jun 20 22:54:24 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.131@KRBTEST.COM
Jun 20 22:54:24 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.130@KRBTEST.COM
I20260620 22:54:24.040905 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.032074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:33237 (local address 127.2.90.130:45007)
0620 22:54:24.032226 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:24.032253 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:24.032280 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:54:24.032421 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:24.032424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:24.032476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:24.032612 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:24.033275 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.034146 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.035105 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.035492 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.035596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:24.037988 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:24.037991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:24.037992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:24.038000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:24.039439 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:24.039799 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:24.039802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:24.039802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:24.039901 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:24.040104 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:24.040106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:24.040106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:24.040339 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:24.040410 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:24.040664 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 22:54:24.040807 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260620 22:54:24.040896 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.031945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42853 (local address 127.2.90.129:50183)
0620 22:54:24.032191 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.032197 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:24.032219 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:24.033017 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:24.033018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:24.033187 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:24.033226 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:24.033227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.034258 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.034259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:24.035031 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:24.035033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.035570 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.035571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:24.035679 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:24.036222 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:24.036226 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:24.037587 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:24.039381 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:24.039388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:24.039390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:24.039731 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:24.040101 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:24.040101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:24.040102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:24.040152 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:24.040473 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:24.040474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:24.040532 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 22:54:24.040602 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 22:54:24.040807 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":66,"thread_start_us":51,"threads_started":1}
I20260620 22:54:24.040999 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.031945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:45007 (local address 127.2.90.129:33237)
0620 22:54:24.032078 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.032088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:24.032142 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:24.032660 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:24.032663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:24.032969 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:24.033169 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:24.033174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.034241 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.034242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:24.035030 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:24.035031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.035572 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.035573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:24.035680 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:24.036174 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:24.036183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:24.037894 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:24.039513 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:24.039514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:24.039514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:24.039732 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:24.039940 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:24.039941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:24.039941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:24.040038 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:24.040403 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:24.040409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:24.040519 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 22:54:24.040602 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:54:24.040878 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":126,"thread_start_us":62,"threads_started":1}
I20260620 22:54:24.041159 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.032076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:50183 (local address 127.2.90.131:42853)
0620 22:54:24.032226 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:24.032253 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:24.032285 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:54:24.032858 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:24.032861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:24.032900 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:24.033018 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:24.033321 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.034169 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.035109 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.035495 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.035587 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:24.037682 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:24.037686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:24.037686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:24.037695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:24.039293 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:24.039969 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:24.039971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:24.039971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:24.040040 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:24.040202 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:24.040203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:24.040203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:24.040412 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:24.040477 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:24.040658 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 22:54:24.040796 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":47,"threads_started":1}
W20260620 22:54:24.041890 17264 leader_election.cc:341] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853): Not found: Tablet not found: 4f51ce18e3f248f08f737fe5b02fe95c
I20260620 22:54:24.041883 17464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f51ce18e3f248f08f737fe5b02fe95c" candidate_uuid: "56bad906eb00437b85726b3409c40eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d2cc2396ef084bba93b850f18a6314ba" is_pre_election: true
I20260620 22:54:24.042032 17464 raft_consensus.cc:2454] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56bad906eb00437b85726b3409c40eba in term 1.
I20260620 22:54:24.042297 17263 leader_election.cc:302] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56bad906eb00437b85726b3409c40eba, d2cc2396ef084bba93b850f18a6314ba; no voters: d619ce7db73e407988ee8112e2543ed4
I20260620 22:54:24.042409 17651 raft_consensus.cc:2781] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 22:54:24.042470 17651 raft_consensus.cc:482] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:54:24.042492 17651 raft_consensus.cc:3037] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:54:24.043579 17651 raft_consensus.cc:504] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:24.043710 17651 leader_election.cc:288] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 2 election: Requested vote from peers d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853), d2cc2396ef084bba93b850f18a6314ba (127.2.90.130:45007)
I20260620 22:54:24.043867 17464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f51ce18e3f248f08f737fe5b02fe95c" candidate_uuid: "56bad906eb00437b85726b3409c40eba" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d2cc2396ef084bba93b850f18a6314ba"
I20260620 22:54:24.043952 17464 raft_consensus.cc:3037] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 1 FOLLOWER]: Advancing to term 2
W20260620 22:54:24.043990 17264 leader_election.cc:341] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853): Not found: Tablet not found: 4f51ce18e3f248f08f737fe5b02fe95c
I20260620 22:54:24.045030 17464 raft_consensus.cc:2454] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56bad906eb00437b85726b3409c40eba in term 2.
I20260620 22:54:24.045172 17263 leader_election.cc:302] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56bad906eb00437b85726b3409c40eba, d2cc2396ef084bba93b850f18a6314ba; no voters: d619ce7db73e407988ee8112e2543ed4
I20260620 22:54:24.045249 17651 raft_consensus.cc:2781] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:54:24.045364 17651 raft_consensus.cc:686] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [term 2 LEADER]: Becoming Leader. State: Replica: 56bad906eb00437b85726b3409c40eba, State: Running, Role: LEADER
I20260620 22:54:24.045436 17651 consensus_queue.cc:237] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:24.045634 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.045542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:34347)
0620 22:54:24.045578 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.045609 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 22:54:24.550542 17464 raft_consensus.cc:1261] T 4f51ce18e3f248f08f737fe5b02fe95c P d2cc2396ef084bba93b850f18a6314ba [term 2 FOLLOWER]: Refusing update from remote peer 56bad906eb00437b85726b3409c40eba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 22:54:24.550868 17658 consensus_queue.cc:1035] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 22:54:24.551671 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.551518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:57491)
0620 22:54:24.551594 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.551633 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":35,"threads_started":1}
I20260620 22:54:24.552471 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.552388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:50121)
0620 22:54:24.552421 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.552444 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 22:54:24.552459 17264 consensus_peers.cc:487] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba -> Peer d619ce7db73e407988ee8112e2543ed4 (127.2.90.131:42853): Couldn't send request to peer d619ce7db73e407988ee8112e2543ed4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f51ce18e3f248f08f737fe5b02fe95c. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:54:24.552556 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.552264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:45775)
0620 22:54:24.552489 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.552523 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260620 22:54:24.668130 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.667923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:57789)
0620 22:54:24.668052 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.668088 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260620 22:54:24.942253 17668 ts_tablet_manager.cc:927] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Initiating tablet copy from peer 56bad906eb00437b85726b3409c40eba (127.2.90.129:37147)
I20260620 22:54:24.942488 17668 tablet_copy_client.cc:316] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: tablet copy: Beginning tablet copy session from remote peer at address 127.2.90.129:37147
Jun 20 22:54:24 dist-test-slave-4mtw krb5kdc[16078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996062, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.129@KRBTEST.COM
I20260620 22:54:24.949502 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.942880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:60881 (local address 127.2.90.129:37147)
0620 22:54:24.943049 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:24.943053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:24.943062 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:24.943084 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:24.943086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:24.943097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:24.943164 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:24.943599 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.944271 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.945079 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.945368 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.945423 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:24.947200 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:24.947207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:24.947207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:24.947214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:24.948374 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:24.948720 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:24.948724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:24.948724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:24.948776 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:24.948930 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:24.948931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:24.948931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:24.949091 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:24.949160 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:24.949318 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 22:54:24.949398 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260620 22:54:24.949514 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.942763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:37147 (local address 127.2.90.131:60881)
0620 22:54:24.942804 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.942816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:24.942888 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:24.943180 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:24.943185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:24.943461 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:24.943527 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:24.943530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.944342 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.944343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:24.945001 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:24.945004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:24.945426 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:24.945427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:24.945503 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:24.945901 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:24.945905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:24.947124 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:24.948444 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:24.948451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:24.948453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:24.948633 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:24.948828 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:24.948828 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:24.948829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:24.948882 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:24.949145 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:24.949152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:24.949204 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 22:54:24.949276 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:54:24.949397 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:54:24.949988 17349 tablet_copy_service.cc:140] P 56bad906eb00437b85726b3409c40eba: Received BeginTabletCopySession request for tablet 4f51ce18e3f248f08f737fe5b02fe95c from peer d619ce7db73e407988ee8112e2543ed4 ({username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:60881)
I20260620 22:54:24.950060 17349 tablet_copy_service.cc:161] P 56bad906eb00437b85726b3409c40eba: Beginning new tablet copy session on tablet 4f51ce18e3f248f08f737fe5b02fe95c from peer d619ce7db73e407988ee8112e2543ed4 at {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:60881: session id = d619ce7db73e407988ee8112e2543ed4-4f51ce18e3f248f08f737fe5b02fe95c
I20260620 22:54:24.950505 17349 tablet_copy_source_session.cc:205] T 4f51ce18e3f248f08f737fe5b02fe95c P 56bad906eb00437b85726b3409c40eba: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 22:54:24.951413 17668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f51ce18e3f248f08f737fe5b02fe95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:24.953867 17668 tablet_copy_client.cc:791] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: tablet copy: Starting download of 0 data blocks...
I20260620 22:54:24.953991 17668 tablet_copy_client.cc:655] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: tablet copy: Starting download of 1 WAL segments...
I20260620 22:54:24.954711 17668 tablet_copy_client.cc:531] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 22:54:24.955336 17668 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Bootstrap starting.
I20260620 22:54:24.956549 17668 log.cc:824] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:24.958444 17668 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:24.958628 17668 tablet_bootstrap.cc:492] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Bootstrap complete.
I20260620 22:54:24.958736 17668 ts_tablet_manager.cc:1397] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 22:54:24.959626 17668 raft_consensus.cc:348] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:24.959766 17668 raft_consensus.cc:729] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d619ce7db73e407988ee8112e2543ed4, State: Initialized, Role: FOLLOWER
I20260620 22:54:24.959862 17668 consensus_queue.cc:260] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d619ce7db73e407988ee8112e2543ed4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42853 } } peers { permanent_uuid: "d2cc2396ef084bba93b850f18a6314ba" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 45007 } } peers { permanent_uuid: "56bad906eb00437b85726b3409c40eba" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 37147 } }
I20260620 22:54:24.960110 17668 ts_tablet_manager.cc:1428] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4: Time spent starting tablet: real 0.001s user 0.001s sys 0.002s
I20260620 22:54:24.960407 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:24.960323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:47949)
0620 22:54:24.960353 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:24.960382 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 22:54:24.960489 17349 tablet_copy_service.cc:342] P 56bad906eb00437b85726b3409c40eba: Request end of tablet copy session d619ce7db73e407988ee8112e2543ed4-4f51ce18e3f248f08f737fe5b02fe95c received from {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:60881
I20260620 22:54:24.960541 17349 tablet_copy_service.cc:433] P 56bad906eb00437b85726b3409c40eba: ending tablet copy session d619ce7db73e407988ee8112e2543ed4-4f51ce18e3f248f08f737fe5b02fe95c on tablet 4f51ce18e3f248f08f737fe5b02fe95c with peer d619ce7db73e407988ee8112e2543ed4
I20260620 22:54:25.519867 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:25.519679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:42433)
0620 22:54:25.519761 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:25.519822 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":39,"threads_started":1}
I20260620 22:54:25.519830 17598 raft_consensus.cc:1203] T 4f51ce18e3f248f08f737fe5b02fe95c P d619ce7db73e407988ee8112e2543ed4 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 22:54:25.520354 17678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:25.520161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:34157)
0620 22:54:25.520253 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:25.520311 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":48,"threads_started":1}
I20260620 22:54:25.553128 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:25.552932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:38313)
0620 22:54:25.553049 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:25.553084 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 22:54:26.520462 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:26.520240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:56915)
0620 22:54:26.520376 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:26.520412 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260620 22:54:26.520867 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:26.520695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:47745)
0620 22:54:26.520788 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:26.520828 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260620 22:54:26.553771 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:26.553544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:52989)
0620 22:54:26.553681 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:26.553723 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260620 22:54:27.521099 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:27.520910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:47147)
0620 22:54:27.521015 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:27.521052 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260620 22:54:27.521394 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:27.521212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:58923)
0620 22:54:27.521326 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:27.521355 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":48,"threads_started":1}
I20260620 22:54:27.521461 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:27.521351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:57387)
0620 22:54:27.521392 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:27.521416 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 22:54:27.554433 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:27.554224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:51285)
0620 22:54:27.554349 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:27.554389 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260620 22:54:28.522084 17687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:28.521876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:58881)
0620 22:54:28.522003 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:28.522038 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260620 22:54:28.522085 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:28.521879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:51953)
0620 22:54:28.522002 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:28.522039 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260620 22:54:28.555054 17688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:28.554837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:43335)
0620 22:54:28.554964 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:28.555005 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260620 22:54:29.522701 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:29.522508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:49307)
0620 22:54:29.522622 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:29.522655 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260620 22:54:29.522701 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:29.522499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:60763)
0620 22:54:29.522624 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:29.522655 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260620 22:54:29.555689 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:29.555506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:35171)
0620 22:54:29.555616 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:29.555645 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260620 22:54:30.523337 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:30.523120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:58487)
0620 22:54:30.523265 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:30.523293 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260620 22:54:30.523343 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:30.523120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:59069)
0620 22:54:30.523262 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:30.523301 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260620 22:54:30.556303 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:30.556114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:58687)
0620 22:54:30.556224 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:30.556259 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260620 22:54:31.523932 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:31.523728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:40149)
0620 22:54:31.523852 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:31.523886 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260620 22:54:31.523954 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:31.523739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:43159)
0620 22:54:31.523876 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:31.523911 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":53,"threads_started":1}
I20260620 22:54:31.556924 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:31.556709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:40399)
0620 22:54:31.556844 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:31.556880 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260620 22:54:32.524508 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:32.524329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:52901)
0620 22:54:32.524440 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:32.524468 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260620 22:54:32.524508 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:32.524316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:51475)
0620 22:54:32.524425 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:32.524461 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260620 22:54:32.557569 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:32.557333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:38511)
0620 22:54:32.557476 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:32.557520 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260620 22:54:33.525098 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:33.524914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:35009)
0620 22:54:33.525023 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:33.525054 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260620 22:54:33.525135 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:33.524949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:37423)
0620 22:54:33.525063 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:33.525094 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260620 22:54:33.558168 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:33.557964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:34975)
0620 22:54:33.558089 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:33.558127 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260620 22:54:34.525784 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:34.525555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:34001)
0620 22:54:34.525688 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:34.525730 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260620 22:54:34.525784 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:34.525555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:33053)
0620 22:54:34.525688 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:34.525730 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260620 22:54:34.558831 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:34.558605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:45573)
0620 22:54:34.558735 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:34.558771 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260620 22:54:35.526429 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:35.526227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:38289)
0620 22:54:35.526340 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:35.526376 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260620 22:54:35.526429 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:35.526228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:39759)
0620 22:54:35.526350 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:35.526379 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":50,"threads_started":1}
I20260620 22:54:35.559382 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:35.559188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:35811)
0620 22:54:35.559308 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:35.559339 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260620 22:54:36.527079 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:36.526865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:40403)
0620 22:54:36.526997 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:36.527030 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260620 22:54:36.527081 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:36.526864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:44953)
0620 22:54:36.526984 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:36.527024 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260620 22:54:36.559952 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:36.559760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:48127)
0620 22:54:36.559874 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:36.559907 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260620 22:54:37.527635 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:37.527449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.131:37307)
0620 22:54:37.527562 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:37.527591 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260620 22:54:37.527742 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:37.527573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.129:52637)
0620 22:54:37.527664 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:37.527694 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260620 22:54:37.560607 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:37.560424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44667 (local address 127.2.90.130:43833)
0620 22:54:37.560541 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:37.560566 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 22:54:37.670019 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 17217
I20260620 22:54:37.674417 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 17378
I20260620 22:54:37.678140 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 17514
2026-06-20T22:54:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18023 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:54:37 dist-test-slave-4mtw krb5kdc[17721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:54:37 dist-test-slave-4mtw krb5kdc[17721](info): set up 2 sockets
Jun 20 22:54:37 dist-test-slave-4mtw krb5kdc[17721](info): commencing operation
krb5kdc: starting...
W20260620 22:54:38.620390 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:54:38 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:54:38.724274 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35411
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:38.801702 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:38.801901 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:38.801931 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:38.803049 17734 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:54:38.803087 17734 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:54:38.803104 17734 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:54:38.803119 17734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:38.803135 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:38.803154 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:38.803177 17734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:38.803193 17734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:38.804615 17734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:38.804865 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:38.805052 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:38.809669 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:38.809644 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:38.809784 17734 server_base.cc:1034] running on GCE node
W20260620 22:54:38.809664 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:38.810027 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:38.810278 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:38.811411 17734 hybrid_clock.cc:648] HybridClock initialized: now 1781996078811398 us; error 27 us; skew 500 ppm
Jun 20 22:54:38 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996078, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:38.813606 17734 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:38.813992 17734 webserver.cc:460] Webserver started at http://127.2.90.190:46827/ using document root <none> and password file <none>
I20260620 22:54:38.814170 17734 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:38.814215 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:38.814332 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:38.815291 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "06c3c0ceb0d8fb8faa79b3bb8dd090cb"
server_key_iv: "fcf731a240f8cca04401396be40cc7a5"
server_key_version: "encryptionkey@0"
I20260620 22:54:38.815591 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "06c3c0ceb0d8fb8faa79b3bb8dd090cb"
server_key_iv: "fcf731a240f8cca04401396be40cc7a5"
server_key_version: "encryptionkey@0"
I20260620 22:54:38.817269 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 22:54:38.817895 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:38.818032 17734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:38.818109 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "06c3c0ceb0d8fb8faa79b3bb8dd090cb"
server_key_iv: "fcf731a240f8cca04401396be40cc7a5"
server_key_version: "encryptionkey@0"
I20260620 22:54:38.818171 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:38.825518 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:38.827091 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:38.827204 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:38.830698 17734 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35411
I20260620 22:54:38.830727 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35411 every 8 connection(s)
I20260620 22:54:38.831019 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:38.831800 17802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:38.833950 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap starting.
I20260620 22:54:38.834486 17802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:38.834736 17802 log.cc:824] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:38.835387 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: No bootstrap required, opened a new log
I20260620 22:54:38.836961 17802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:38.837092 17802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:38.837124 17802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0eb5c83a854b7dabdab9ef6793f5d2, State: Initialized, Role: FOLLOWER
I20260620 22:54:38.837260 17802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:38.837343 17802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:38.837378 17802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:38.837425 17802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:38.837913 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17734
I20260620 22:54:38.837929 17802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:38.838003 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:54:38.838013 17802 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b0eb5c83a854b7dabdab9ef6793f5d2; no voters:
I20260620 22:54:38.838171 17802 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:54:38.838186 2410 external_mini_cluster.cc:1428] Setting key 2ce9eae49af2d1a580539991a7fabae1
I20260620 22:54:38.838205 17805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:38.838315 17805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 0b0eb5c83a854b7dabdab9ef6793f5d2, State: Running, Role: LEADER
I20260620 22:54:38.838424 17802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:38.838444 17805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:38.839126 17806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } } }
I20260620 22:54:38.839202 17806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:38.839411 17807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0eb5c83a854b7dabdab9ef6793f5d2. Latest consensus state: current_term: 1 leader_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } } }
I20260620 22:54:38.839532 17807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: This master's current role is: LEADER
W20260620 22:54:38.840217 17823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:54:38.840271 17823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:54:38.840312 17816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:38.840457 17816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:38.841862 17816 catalog_manager.cc:1347] Generated new cluster ID: f4a58789ffc64b45928732feeaf8bea2
I20260620 22:54:38.841908 17816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:54:38 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:38.849174 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:38.838615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.2.90.190:35411)
0620 22:54:38.838805 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:38.838814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:38.838836 (+ 22us) server_negotiation.cc:407] Connection header received
0620 22:54:38.838928 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:38.838935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:38.839004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:38.839990 (+ 986us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:38.840094 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:38.841318 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:38.842395 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:38.842972 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:38.843063 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:38.845688 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:38.845693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:38.845693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:38.845706 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:38.847532 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:38.847878 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:38.847881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:38.847881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:38.847991 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:38.848174 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:38.848175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:38.848175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:38.848522 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:38.848658 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:38.848824 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 22:54:38.848962 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260620 22:54:38.852239 17816 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:54:38.852942 17816 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:38.867157 17816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Generated new TSK 0
I20260620 22:54:38.867322 17816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:54:38.893499 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35411
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:38.969743 17828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:38.969930 17828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:38.969959 17828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:38.971058 17828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:38.971094 17828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:38.971118 17828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:38.972525 17828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:38.972756 17828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:38.972975 17828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:38.977478 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:38.977430 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:38.977430 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:38.977656 17828 server_base.cc:1034] running on GCE node
I20260620 22:54:38.977865 17828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:38.978068 17828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:38.979205 17828 hybrid_clock.cc:648] HybridClock initialized: now 1781996078979195 us; error 26 us; skew 500 ppm
Jun 20 22:54:38 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996078, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:38.981426 17828 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:38.981812 17828 webserver.cc:460] Webserver started at http://127.2.90.129:32805/ using document root <none> and password file <none>
I20260620 22:54:38.982018 17828 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:38.982060 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:38.982175 17828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:38.983111 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "e2366f0a3b3f4e7e90386c590e2a3989"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "858700cf71434e38e50a4cb6800d97cb"
server_key_iv: "30daaf2943c7d0a382a8fc6103185c44"
server_key_version: "encryptionkey@0"
I20260620 22:54:38.983402 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "e2366f0a3b3f4e7e90386c590e2a3989"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "858700cf71434e38e50a4cb6800d97cb"
server_key_iv: "30daaf2943c7d0a382a8fc6103185c44"
server_key_version: "encryptionkey@0"
I20260620 22:54:38.984527 17828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:54:38.985215 17843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:38.985354 17828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:38.985428 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "e2366f0a3b3f4e7e90386c590e2a3989"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "858700cf71434e38e50a4cb6800d97cb"
server_key_iv: "30daaf2943c7d0a382a8fc6103185c44"
server_key_version: "encryptionkey@0"
I20260620 22:54:38.985488 17828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:38.994690 17828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:38.996275 17828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:38.996395 17828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:38.996573 17828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:38.996863 17828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:38.996896 17828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:38.996920 17828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:38.996953 17828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.002259 17828 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:46831
I20260620 22:54:39.002319 17956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:46831 every 8 connection(s)
I20260620 22:54:39.002574 17828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:54:39.007645 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17828
I20260620 22:54:39.007786 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:54:39.007995 2410 external_mini_cluster.cc:1428] Setting key afad2ae55b696412cf20669caa27bde1
Jun 20 22:54:39 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996078, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:39.010938 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.003192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:42713 (local address 127.2.90.190:35411)
0620 22:54:39.003253 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.003255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.003391 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:54:39.003820 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.003821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.003829 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.003876 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:39.004721 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.005231 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.006161 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.006358 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.006412 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.008504 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:39.008505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:39.008506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:39.008514 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:39.009778 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.010155 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.010156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.010156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.010214 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.010398 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.010398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.010399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.010517 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:39.010578 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.010781 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.010857 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:54:39.011020 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.003138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:42713)
0620 22:54:39.003281 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.003319 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.003752 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.003932 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.003935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.004438 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:39.004623 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.004628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.005302 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.005303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.006082 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.006084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.006425 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.006426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.006559 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.006960 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:39.006967 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:39.008416 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:39.009864 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:39.009872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:39.009874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:39.010093 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:39.010270 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:39.010270 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:39.010270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:39.010354 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:39.010569 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:39.010572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:39.010673 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.010751 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.010861 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260620 22:54:39.011514 17957 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35411
I20260620 22:54:39.011602 17957 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:39.011770 17957 heartbeater.cc:511] Master 127.2.90.190:35411 requested a full tablet report, sending...
I20260620 22:54:39.012290 17766 ts_manager.cc:194] Registered new tserver with Master: e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831)
I20260620 22:54:39.013180 17766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:42713
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:54:39.028889 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35411
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:39.105825 17962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:39.106029 17962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:39.106060 17962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:39.107201 17962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:39.107241 17962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:39.107276 17962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:54:39.108700 17962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:39.108927 17962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:39.109118 17962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:39.113770 17970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:39.113803 17962 server_base.cc:1034] running on GCE node
W20260620 22:54:39.113770 17967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:39.113766 17968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:39.114192 17962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:39.114440 17962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:39.115573 17962 hybrid_clock.cc:648] HybridClock initialized: now 1781996079115562 us; error 29 us; skew 500 ppm
Jun 20 22:54:39 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996079, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:39.117815 17962 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:54:39.118222 17962 webserver.cc:460] Webserver started at http://127.2.90.130:44495/ using document root <none> and password file <none>
I20260620 22:54:39.118431 17962 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:39.118474 17962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:39.118588 17962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:39.119585 17962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "72683c48431b4a65a329d2b47f47a180"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "1013835c5a876a97cd2a1359077355e0"
server_key_iv: "543d4b284c9de54f7b89951c1565db6f"
server_key_version: "encryptionkey@0"
I20260620 22:54:39.119887 17962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "72683c48431b4a65a329d2b47f47a180"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "1013835c5a876a97cd2a1359077355e0"
server_key_iv: "543d4b284c9de54f7b89951c1565db6f"
server_key_version: "encryptionkey@0"
I20260620 22:54:39.121079 17962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:54:39.121798 17977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.121951 17962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.122022 17962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "72683c48431b4a65a329d2b47f47a180"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "1013835c5a876a97cd2a1359077355e0"
server_key_iv: "543d4b284c9de54f7b89951c1565db6f"
server_key_version: "encryptionkey@0"
I20260620 22:54:39.122082 17962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:39.140954 17962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:39.142688 17962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:39.142800 17962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:39.143014 17962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:39.143294 17962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:39.143326 17962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.143358 17962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:39.143380 17962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.148638 17962 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:37199
I20260620 22:54:39.148710 18090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:37199 every 8 connection(s)
I20260620 22:54:39.149004 17962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:54:39.153523 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 17962
I20260620 22:54:39.153627 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:54:39.153834 2410 external_mini_cluster.cc:1428] Setting key 3a39a97670ad40bde70039732d597fca
Jun 20 22:54:39 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996079, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:39.157920 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.149718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:36471 (local address 127.2.90.190:35411)
0620 22:54:39.149826 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.149829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.149948 (+ 119us) server_negotiation.cc:407] Connection header received
0620 22:54:39.150456 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.150457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.150471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.150518 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:39.151458 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.151947 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.152954 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.153147 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.153201 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.155388 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:39.155390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:39.155390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:39.155398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:39.156615 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.157029 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.157030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.157030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.157086 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.157296 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.157297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.157297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.157413 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:39.157469 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.157724 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.157817 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:54:39.158057 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.149643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:36471)
0620 22:54:39.149820 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.149869 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.150358 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.150596 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.150600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.151171 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:39.151365 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.151371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.152028 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.152029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.152874 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.152877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.153212 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.153213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.153365 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.153875 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:39.153885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:39.155306 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:39.156709 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:39.156716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:39.156718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:39.156950 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:39.157139 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:39.157139 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:39.157140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:39.157242 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:39.157468 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:39.157474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:39.157607 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.157695 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.157847 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260620 22:54:39.158469 18091 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35411
I20260620 22:54:39.158556 18091 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:39.158725 18091 heartbeater.cc:511] Master 127.2.90.190:35411 requested a full tablet report, sending...
I20260620 22:54:39.159081 17766 ts_manager.cc:194] Registered new tserver with Master: 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
I20260620 22:54:39.159592 17766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:36471
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:54:39.177309 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35411
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:39.255146 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:39.255331 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:39.255352 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:39.256424 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:39.256458 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:39.256484 18096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:54:39.257905 18096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:39.258085 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:39.258255 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:39.262629 18101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:39.262684 18102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:39.262856 18096 server_base.cc:1034] running on GCE node
W20260620 22:54:39.262909 18104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:39.263109 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:39.263294 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:39.264421 18096 hybrid_clock.cc:648] HybridClock initialized: now 1781996079264411 us; error 28 us; skew 500 ppm
Jun 20 22:54:39 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996079, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:39.266722 18096 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:54:39.267098 18096 webserver.cc:460] Webserver started at http://127.2.90.131:34213/ using document root <none> and password file <none>
I20260620 22:54:39.267293 18096 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:39.267334 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:39.267464 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:54:39.268400 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "d7637a8f607644cbafc8d0e84351679a"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "a0779ef7922051b7dcdac835398b8fd9"
server_key_iv: "21f60a24425859d321341017c12c707f"
server_key_version: "encryptionkey@0"
I20260620 22:54:39.268750 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "d7637a8f607644cbafc8d0e84351679a"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "a0779ef7922051b7dcdac835398b8fd9"
server_key_iv: "21f60a24425859d321341017c12c707f"
server_key_version: "encryptionkey@0"
I20260620 22:54:39.269977 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:39.270608 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.270751 18096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:39.270824 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d7637a8f607644cbafc8d0e84351679a"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "a0779ef7922051b7dcdac835398b8fd9"
server_key_iv: "21f60a24425859d321341017c12c707f"
server_key_version: "encryptionkey@0"
I20260620 22:54:39.270891 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:39.279748 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:39.281370 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:39.281494 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:39.281669 18096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:39.281934 18096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:54:39.281970 18096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.281996 18096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:54:39.282018 18096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:39.287137 18096 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42747
I20260620 22:54:39.287176 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42747 every 8 connection(s)
I20260620 22:54:39.287461 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:54:39.291126 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18096
I20260620 22:54:39.291224 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:54:39.291396 2410 external_mini_cluster.cc:1428] Setting key 8a5db4ddb80a7b9df6f0e21f13a1a5f3
Jun 20 22:54:39 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996079, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:39.295679 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.288120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:48351 (local address 127.2.90.190:35411)
0620 22:54:39.288176 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.288178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.288344 (+ 166us) server_negotiation.cc:407] Connection header received
0620 22:54:39.288845 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.288846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.288858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.288907 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:39.289766 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.290257 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.291191 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.291431 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.291490 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.293549 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:39.293550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:39.293551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:39.293557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:39.294709 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.295036 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.295036 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.295037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.295080 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.295244 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.295245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.295245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.295336 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:39.295378 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.295567 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.295619 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:54:39.295845 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.288067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:48351)
0620 22:54:39.288233 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.288271 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.288752 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.288971 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.288974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.289500 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:39.289675 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.289680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.290334 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.290335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.291111 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.291113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.291499 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.291500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.291632 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.292136 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:39.292145 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:39.293474 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:39.294780 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:39.294785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:39.294786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:39.294981 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:39.295122 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:39.295123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:39.295123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:39.295204 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:39.295374 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:39.295376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:39.295476 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.295547 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.295652 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260620 22:54:39.296239 18225 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35411
I20260620 22:54:39.296319 18225 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:39.296474 18225 heartbeater.cc:511] Master 127.2.90.190:35411 requested a full tablet report, sending...
I20260620 22:54:39.296841 17766 ts_manager.cc:194] Registered new tserver with Master: d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747)
I20260620 22:54:39.297168 17766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:48351
I20260620 22:54:39.302811 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:54:39.308928 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.304028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.2.90.190:35411)
0620 22:54:39.304069 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.304071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.304168 (+ 97us) server_negotiation.cc:407] Connection header received
0620 22:54:39.304215 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.304215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.304223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.304256 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:39.304748 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.305205 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.305863 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.306035 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.306073 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.306999 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:39.307000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:39.307001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:39.307006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:39.308111 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.308363 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.308363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.308364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.308406 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:39.308519 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:39.308519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:39.308519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:39.308634 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:39.308728 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.308817 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.308870 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:54:39.310328 17766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:54:39.311759 17766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:54:39.317310 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.313571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37199 (local address 127.0.0.1:54272)
0620 22:54:39.313839 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.313847 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.313886 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.314111 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.314114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.314117 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:39.314177 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.314179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.315526 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.315527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.316284 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.316287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.317110 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.317111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.317163 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.317174 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.317224 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.317258 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260620 22:54:39.317360 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.313691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.2.90.130:37199)
0620 22:54:39.313981 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.313986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.313995 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:39.314023 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.314025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.314036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.314100 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:39.314263 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.315432 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.316365 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.317040 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.317112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.317166 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.317236 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.317285 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":66,"threads_started":1}
I20260620 22:54:39.317974 18025 tablet_service.cc:1467] Processing CreateTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 (DEFAULT_TABLE table=test-table [id=033e76031ffb42dea3b8b130a6382248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:39.318315 18025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26099f8d7cd4622bb0c4637d357bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:39.318323 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.313566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42747 (local address 127.0.0.1:51804)
0620 22:54:39.313743 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.313782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.313846 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.314500 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.314501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.314502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:39.314538 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.314539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.315370 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.315371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.316617 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.316620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.318101 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.318103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.318164 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.318168 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.318219 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.318259 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":6,"thread_start_us":71,"threads_started":1}
I20260620 22:54:39.318425 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.313770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46831 (local address 127.0.0.1:51548)
0620 22:54:39.313873 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.313883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.313929 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.314268 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.314269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.314270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:39.314298 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.314300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.315492 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.315493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.316617 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.316620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.318177 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.318178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.318241 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.318246 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.318272 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.318304 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260620 22:54:39.318537 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.313651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.2.90.131:42747)
0620 22:54:39.314233 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.314237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.314246 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:39.314279 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.314281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.314399 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.314619 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:39.314669 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.315285 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.317502 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.318033 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.318083 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.318119 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.318422 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.318466 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":82,"threads_started":1}
I20260620 22:54:39.318748 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.313976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.2.90.129:46831)
0620 22:54:39.314118 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.314122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.314131 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:54:39.314165 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.314166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.314177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.314271 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:39.314350 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.315404 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.316717 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.317419 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.318357 (+ 938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.318427 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.318452 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.318674 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260620 22:54:39.319110 18159 tablet_service.cc:1467] Processing CreateTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 (DEFAULT_TABLE table=test-table [id=033e76031ffb42dea3b8b130a6382248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:39.319262 17891 tablet_service.cc:1467] Processing CreateTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 (DEFAULT_TABLE table=test-table [id=033e76031ffb42dea3b8b130a6382248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:54:39.319412 18159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26099f8d7cd4622bb0c4637d357bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:39.319489 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26099f8d7cd4622bb0c4637d357bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:54:39.321938 18240 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap starting.
I20260620 22:54:39.322203 18239 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Bootstrap starting.
I20260620 22:54:39.322400 18241 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Bootstrap starting.
I20260620 22:54:39.322603 18240 tablet_bootstrap.cc:654] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:39.322772 18239 tablet_bootstrap.cc:654] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:39.322979 18241 tablet_bootstrap.cc:654] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Neither blocks nor log segments found. Creating new log.
I20260620 22:54:39.323210 18240 log.cc:824] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:39.323262 18239 log.cc:824] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:39.323421 18241 log.cc:824] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:39.323945 18240 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: No bootstrap required, opened a new log
I20260620 22:54:39.324008 18240 ts_tablet_manager.cc:1397] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:54:39.324087 18239 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: No bootstrap required, opened a new log
I20260620 22:54:39.324092 18241 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: No bootstrap required, opened a new log
I20260620 22:54:39.324153 18239 ts_tablet_manager.cc:1397] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:54:39.324153 18241 ts_tablet_manager.cc:1397] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:54:39.325630 18240 raft_consensus.cc:348] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.325763 18240 raft_consensus.cc:374] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:39.325795 18240 raft_consensus.cc:729] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2366f0a3b3f4e7e90386c590e2a3989, State: Initialized, Role: FOLLOWER
I20260620 22:54:39.325903 18240 consensus_queue.cc:260] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.326138 18240 ts_tablet_manager.cc:1428] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:54:39.326036 18241 raft_consensus.cc:348] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.326102 18239 raft_consensus.cc:348] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.326190 18241 raft_consensus.cc:374] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:39.326225 18239 raft_consensus.cc:374] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:54:39.326223 18241 raft_consensus.cc:729] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7637a8f607644cbafc8d0e84351679a, State: Initialized, Role: FOLLOWER
I20260620 22:54:39.326254 18239 raft_consensus.cc:729] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72683c48431b4a65a329d2b47f47a180, State: Initialized, Role: FOLLOWER
I20260620 22:54:39.326145 17957 heartbeater.cc:503] Master 127.2.90.190:35411 was elected leader, sending a full tablet report...
I20260620 22:54:39.326371 18239 consensus_queue.cc:260] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.326426 18241 consensus_queue.cc:260] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.326645 18091 heartbeater.cc:503] Master 127.2.90.190:35411 was elected leader, sending a full tablet report...
I20260620 22:54:39.326661 18241 ts_tablet_manager.cc:1428] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:54:39.326774 18225 heartbeater.cc:503] Master 127.2.90.190:35411 was elected leader, sending a full tablet report...
I20260620 22:54:39.326912 18239 ts_tablet_manager.cc:1428] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260620 22:54:39.399531 18092 tablet.cc:2369] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:54:39.503232 17958 tablet.cc:2369] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:54:39.533064 18246 raft_consensus.cc:482] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:54:39.533203 18246 raft_consensus.cc:504] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.533542 18246 leader_election.cc:288] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831)
I20260620 22:54:39.536559 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.533707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42747 (local address 127.2.90.130:42397)
0620 22:54:39.533760 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.533776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.533839 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.533977 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.533978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.533980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:39.534039 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.534040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.534778 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.534779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.535603 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.535605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.536271 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.536272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.536375 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.536380 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.536440 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.536483 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:54:39.536551 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.533802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:42397 (local address 127.2.90.131:42747)
0620 22:54:39.533847 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.533849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.533861 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:54:39.533888 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.533889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.533897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.533961 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:39.534139 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.534704 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.535702 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.536193 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.536244 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.536252 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.536448 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.536479 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:54:39.536801 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.533884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:36553 (local address 127.2.90.129:46831)
0620 22:54:39.533928 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.533930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.534111 (+ 181us) server_negotiation.cc:407] Connection header received
0620 22:54:39.534203 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.534204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.534217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.534250 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:39.534414 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.534932 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.536017 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.536433 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.536494 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.536502 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.536669 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.536739 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:54:39.536989 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.533826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46831 (local address 127.2.90.130:36553)
0620 22:54:39.533979 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.533988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:39.534132 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:39.534288 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:39.534289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:39.534291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:39.534361 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.534362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.535083 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.535084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.535928 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:39.535930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.536504 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.536505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:39.536574 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.536578 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:54:39.536883 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 22:54:39.536929 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260620 22:54:39.536947 18179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26099f8d7cd4622bb0c4637d357bf07" candidate_uuid: "72683c48431b4a65a329d2b47f47a180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7637a8f607644cbafc8d0e84351679a" is_pre_election: true
I20260620 22:54:39.537111 18179 raft_consensus.cc:2454] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72683c48431b4a65a329d2b47f47a180 in term 0.
I20260620 22:54:39.537303 17911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26099f8d7cd4622bb0c4637d357bf07" candidate_uuid: "72683c48431b4a65a329d2b47f47a180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" is_pre_election: true
I20260620 22:54:39.537338 17978 leader_election.cc:302] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a; no voters:
I20260620 22:54:39.537457 17911 raft_consensus.cc:2454] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72683c48431b4a65a329d2b47f47a180 in term 0.
I20260620 22:54:39.537462 18246 raft_consensus.cc:2781] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:54:39.537516 18246 raft_consensus.cc:482] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:54:39.537546 18246 raft_consensus.cc:3037] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Advancing to term 1
W20260620 22:54:39.537961 18226 tablet.cc:2369] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:54:39.538542 18246 raft_consensus.cc:504] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.538659 18246 leader_election.cc:288] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [CANDIDATE]: Term 1 election: Requested vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831)
I20260620 22:54:39.538867 17911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26099f8d7cd4622bb0c4637d357bf07" candidate_uuid: "72683c48431b4a65a329d2b47f47a180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2366f0a3b3f4e7e90386c590e2a3989"
I20260620 22:54:39.538856 18179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26099f8d7cd4622bb0c4637d357bf07" candidate_uuid: "72683c48431b4a65a329d2b47f47a180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7637a8f607644cbafc8d0e84351679a"
I20260620 22:54:39.538935 18179 raft_consensus.cc:3037] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:39.538936 17911 raft_consensus.cc:3037] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:54:39.539798 18179 raft_consensus.cc:2454] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72683c48431b4a65a329d2b47f47a180 in term 1.
I20260620 22:54:39.539794 17911 raft_consensus.cc:2454] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72683c48431b4a65a329d2b47f47a180 in term 1.
I20260620 22:54:39.539978 17978 leader_election.cc:302] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a; no voters:
I20260620 22:54:39.540064 18246 raft_consensus.cc:2781] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:54:39.540216 18246 raft_consensus.cc:686] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 1 LEADER]: Becoming Leader. State: Replica: 72683c48431b4a65a329d2b47f47a180, State: Running, Role: LEADER
I20260620 22:54:39.540313 18246 consensus_queue.cc:237] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:39.541069 17765 catalog_manager.cc:5495] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72683c48431b4a65a329d2b47f47a180 (127.2.90.130). New cstate: current_term: 1 leader_uuid: "72683c48431b4a65a329d2b47f47a180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } health_report { overall_health: HEALTHY } } }
I20260620 22:54:39.598650 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.596647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.2.90.130:37199)
0620 22:54:39.596715 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:39.596717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:39.596725 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:54:39.596760 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:39.596761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:39.596784 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:39.596842 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:54:39.596940 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.597396 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.598069 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:39.598240 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:39.598280 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:39.598390 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:54:39.598493 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:54:39.598546 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:39.598560 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 22:54:39.598576 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 22:54:39.600350 18179 raft_consensus.cc:1261] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Refusing update from remote peer 72683c48431b4a65a329d2b47f47a180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:39.600351 17911 raft_consensus.cc:1261] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Refusing update from remote peer 72683c48431b4a65a329d2b47f47a180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:54:39.600693 18246 consensus_queue.cc:1035] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:39.600847 18249 consensus_queue.cc:1035] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:54:39.603157 18253 mvcc.cc:205] Tried to move back new op lower bound from 7299055942040530944 to 7299055941797654528. Current Snapshot: MvccSnapshot[applied={T|T < 7299055942040530944}]
I20260620 22:54:39.605670 17766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:39.605746 17766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:54:39.606405 17766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Sending DeleteTablet for 3 replicas of tablet b26099f8d7cd4622bb0c4637d357bf07
I20260620 22:54:39.606688 18025 tablet_service.cc:1514] Processing DeleteTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:39 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:54272
I20260620 22:54:39.606775 18159 tablet_service.cc:1514] Processing DeleteTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:39 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:51804
I20260620 22:54:39.606784 17891 tablet_service.cc:1514] Processing DeleteTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:39 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:51548
I20260620 22:54:39.606889 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 17828
I20260620 22:54:39.606993 18261 tablet_replica.cc:330] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: stopping tablet replica
I20260620 22:54:39.607081 18261 raft_consensus.cc:2229] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:54:39.607188 18261 raft_consensus.cc:2258] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:39.607685 18261 ts_tablet_manager.cc:1905] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:39.609539 18261 ts_tablet_manager.cc:1918] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:39.609665 18261 log.cc:1196] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/b26099f8d7cd4622bb0c4637d357bf07
I20260620 22:54:39.609985 18261 ts_tablet_manager.cc:1939] T b26099f8d7cd4622bb0c4637d357bf07 P 72683c48431b4a65a329d2b47f47a180: Deleting consensus metadata
I20260620 22:54:39.610445 17751 catalog_manager.cc:4841] TS 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): tablet b26099f8d7cd4622bb0c4637d357bf07 (table test-table [id=033e76031ffb42dea3b8b130a6382248]) successfully deleted
W20260620 22:54:39.611543 17750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:46831 (error 108)
I20260620 22:54:39.611917 18262 tablet_replica.cc:330] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: stopping tablet replica
I20260620 22:54:39.611975 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:39.611843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46831 (local address 127.0.0.1:51556)
0620 22:54:39.611904 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:39.611945 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 22:54:39.612031 18262 raft_consensus.cc:2229] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 22:54:39.612068 17750 catalog_manager.cc:4584] TS e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b26099f8d7cd4622bb0c4637d357bf07: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46831: connect: Connection refused (error 111)
I20260620 22:54:39.612133 18262 raft_consensus.cc:2258] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:39.612320 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 17962
I20260620 22:54:39.612517 18262 ts_tablet_manager.cc:1905] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:39.613801 18262 ts_tablet_manager.cc:1918] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:39.613854 18262 log.cc:1196] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/b26099f8d7cd4622bb0c4637d357bf07
I20260620 22:54:39.614029 18262 ts_tablet_manager.cc:1939] T b26099f8d7cd4622bb0c4637d357bf07 P d7637a8f607644cbafc8d0e84351679a: Deleting consensus metadata
I20260620 22:54:39.614333 17750 catalog_manager.cc:4841] TS d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): tablet b26099f8d7cd4622bb0c4637d357bf07 (table test-table [id=033e76031ffb42dea3b8b130a6382248]) successfully deleted
I20260620 22:54:39.617334 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 18096
I20260620 22:54:39.621841 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 17734
I20260620 22:54:59.626310 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35411
--webserver_interface=127.2.90.190
--webserver_port=46827
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:54:59.704042 18265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:59.704237 18265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:59.704265 18265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:59.705404 18265 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:54:59.705446 18265 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:54:59.705463 18265 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:54:59.705477 18265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:54:59.705492 18265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:59.705509 18265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:59.705529 18265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:54:59.705549 18265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:54:59.706969 18265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:59.707203 18265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:59.707398 18265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:59.715473 18274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:59.715509 18271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:59.715476 18272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:59.715500 18265 server_base.cc:1034] running on GCE node
I20260620 22:54:59.715983 18265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:59.716282 18265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:59.717408 18265 hybrid_clock.cc:648] HybridClock initialized: now 1781996099717387 us; error 38 us; skew 500 ppm
Jun 20 22:54:59 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996099, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:59.722834 18265 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:54:59.723269 18265 webserver.cc:460] Webserver started at http://127.2.90.190:46827/ using document root <none> and password file <none>
I20260620 22:54:59.723474 18265 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:59.723536 18265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:59.724844 18265 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:59.725425 18281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:59.725570 18265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:59.725659 18265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "06c3c0ceb0d8fb8faa79b3bb8dd090cb"
server_key_iv: "fcf731a240f8cca04401396be40cc7a5"
server_key_version: "encryptionkey@0"
I20260620 22:54:59.726050 18265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:59.756129 18265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:59.758261 18265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:59.758394 18265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:59.762156 18265 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35411
I20260620 22:54:59.762190 18333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35411 every 8 connection(s)
I20260620 22:54:59.762545 18265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:54:59.763927 18334 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:54:59.764520 18334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap starting.
I20260620 22:54:59.771090 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18265
I20260620 22:54:59.771435 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:46831
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:35411
--webserver_port=32805
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:59.773025 18334 log.cc:824] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:59.774803 18334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:59.775071 18334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap complete.
I20260620 22:54:59.777148 18334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:59.777365 18334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0eb5c83a854b7dabdab9ef6793f5d2, State: Initialized, Role: FOLLOWER
I20260620 22:54:59.777503 18334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:59.777560 18334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:54:59.777602 18334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:54:59.777647 18334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 22:54:59.778581 18334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:59.778664 18334 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b0eb5c83a854b7dabdab9ef6793f5d2; no voters:
I20260620 22:54:59.778803 18334 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 22:54:59.778863 18338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 22:54:59.779065 18338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 LEADER]: Becoming Leader. State: Replica: 0b0eb5c83a854b7dabdab9ef6793f5d2, State: Running, Role: LEADER
I20260620 22:54:59.779050 18334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:54:59.779186 18338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:54:59.780530 18339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } } }
I20260620 22:54:59.780695 18340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0eb5c83a854b7dabdab9ef6793f5d2. Latest consensus state: current_term: 2 leader_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } } }
I20260620 22:54:59.780773 18339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:59.780789 18340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:54:59.781052 18353 catalog_manager.cc:1259] Loaded cluster ID: f4a58789ffc64b45928732feeaf8bea2
I20260620 22:54:59.781085 18353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: loading cluster ID for follower catalog manager: success
I20260620 22:54:59.781870 18353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: acquiring CA information for follower catalog manager: success
I20260620 22:54:59.782191 18353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:54:59.782275 18355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:54:59.782600 18355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:54:59.782666 18355 catalog_manager.cc:1259] Loaded cluster ID: f4a58789ffc64b45928732feeaf8bea2
I20260620 22:54:59.782721 18355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:54:59.782892 18355 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:54:59.782963 18355 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Loaded TSK: 0
I20260620 22:54:59.788985 18355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Generated new TSK 1
I20260620 22:54:59.789062 18355 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:54:59.878801 18336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:54:59.879035 18336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:54:59.879067 18336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:54:59.880394 18336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:54:59.880440 18336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:54:59.880476 18336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:54:59.881963 18336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:46831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=32805
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:54:59.882185 18336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:54:59.882409 18336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:54:59.887097 18362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:54:59.887138 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:59.887243 18336 server_base.cc:1034] running on GCE node
W20260620 22:54:59.887136 18364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:54:59.887523 18336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:54:59.887746 18336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:54:59.888883 18336 hybrid_clock.cc:648] HybridClock initialized: now 1781996099888870 us; error 32 us; skew 500 ppm
Jun 20 22:54:59 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996099, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:54:59.891095 18336 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:54:59.891510 18336 webserver.cc:460] Webserver started at http://127.2.90.129:32805/ using document root <none> and password file <none>
I20260620 22:54:59.891716 18336 fs_manager.cc:359] Metadata directory not provided
I20260620 22:54:59.891768 18336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:54:59.893046 18336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:54:59.893661 18371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:54:59.893826 18336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:54:59.893904 18336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "e2366f0a3b3f4e7e90386c590e2a3989"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "858700cf71434e38e50a4cb6800d97cb"
server_key_iv: "30daaf2943c7d0a382a8fc6103185c44"
server_key_version: "encryptionkey@0"
I20260620 22:54:59.894210 18336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:54:59.921401 18336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:54:59.923421 18336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:54:59.923569 18336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:54:59.923780 18336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:54:59.924197 18378 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:54:59.925594 18336 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:54:59.925640 18336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:54:59.925663 18336 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:54:59.926283 18336 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:54:59.926323 18336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:54:59.926376 18378 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap starting.
I20260620 22:54:59.933583 18336 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:46831
I20260620 22:54:59.934038 18336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:54:59.934417 18485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:46831 every 8 connection(s)
I20260620 22:54:59.936388 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18336
I20260620 22:54:59.936746 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:37199
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:35411
--webserver_port=44495
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:54:59.937536 18378 log.cc:824] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Log is configured to *not* fsync() on all Append() calls
I20260620 22:54:59.939558 18378 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:54:59.939858 18378 tablet_bootstrap.cc:492] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap complete.
I20260620 22:54:59.940800 18378 ts_tablet_manager.cc:1397] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
Jun 20 22:54:59 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996099, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:54:59.942014 18378 raft_consensus.cc:348] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:59.942309 18378 raft_consensus.cc:729] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2366f0a3b3f4e7e90386c590e2a3989, State: Initialized, Role: FOLLOWER
I20260620 22:54:59.942467 18378 consensus_queue.cc:260] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } }
I20260620 22:54:59.942710 18378 ts_tablet_manager.cc:1428] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 22:54:59.944423 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:59.934829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:37763 (local address 127.2.90.190:35411)
0620 22:54:59.935034 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:59.935044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:59.935081 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:54:59.935598 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:59.935609 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:59.935694 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:59.935862 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:54:59.936453 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.937480 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.938634 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.939021 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.939114 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:59.941319 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:54:59.941323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:54:59.941323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:54:59.941333 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:54:59.942901 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:59.943262 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:59.943264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:59.943265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:59.943357 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:54:59.943573 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:54:59.943573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:54:59.943574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:54:59.943788 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:54:59.943857 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:59.944126 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 22:54:59.944246 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260620 22:54:59.944422 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:59.934725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:37763)
0620 22:54:59.934909 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:59.934958 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:59.935391 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:59.935879 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:59.935882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:59.936142 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:54:59.936326 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:59.936331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.937592 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.937594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:59.938551 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:59.938554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.939112 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.939113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:59.939253 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:59.939754 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:54:59.939764 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:54:59.941224 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:54:59.942972 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:59.942978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:59.942980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:59.943192 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:59.943413 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:54:59.943414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:54:59.943414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:54:59.943518 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:54:59.943847 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:54:59.943853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:54:59.943980 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 22:54:59.944065 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:54:59.944233 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260620 22:54:59.945111 18486 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35411
I20260620 22:54:59.945207 18486 heartbeater.cc:464] Registering TS with master...
I20260620 22:54:59.945384 18486 heartbeater.cc:511] Master 127.2.90.190:35411 requested a full tablet report, sending...
I20260620 22:54:59.946112 18298 ts_manager.cc:194] Registered new tserver with Master: e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831)
I20260620 22:54:59.947130 18298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:37763
I20260620 22:54:59.948040 18486 heartbeater.cc:503] Master 127.2.90.190:35411 was elected leader, sending a full tablet report...
I20260620 22:54:59.955533 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:59.946650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46831 (local address 127.0.0.1:59882)
0620 22:54:59.952770 (+ 6120us) negotiation.cc:107] Waiting for socket to connect
0620 22:54:59.952800 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 22:54:59.952866 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:54:59.953046 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:54:59.953049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:54:59.953051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:54:59.953114 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:59.953116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.953877 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.953878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:59.954665 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:54:59.954667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.955254 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.955255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:54:59.955334 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:59.955346 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 22:54:59.955402 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:54:59.955459 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6095,"thread_start_us":73,"threads_started":1}
I20260620 22:54:59.955529 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:54:59.946721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.2.90.129:46831)
0620 22:54:59.946860 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 22:54:59.946864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:54:59.952862 (+ 5998us) server_negotiation.cc:407] Connection header received
0620 22:54:59.952964 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:54:59.952967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:54:59.952980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:54:59.953039 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:54:59.953169 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.953807 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.954738 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:54:59.955183 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:54:59.955242 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:54:59.955294 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:54:59.955421 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 22:54:59.955459 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260620 22:54:59.955924 18420 tablet_service.cc:1514] Processing DeleteTablet for tablet b26099f8d7cd4622bb0c4637d357bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:54:39 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:59882
I20260620 22:54:59.956488 18495 tablet_replica.cc:330] stopping tablet replica
I20260620 22:54:59.956581 18495 raft_consensus.cc:2229] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:54:59.956647 18495 raft_consensus.cc:2258] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:54:59.956931 18495 ts_tablet_manager.cc:1905] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:54:59.958763 18495 ts_tablet_manager.cc:1918] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:54:59.958822 18495 log.cc:1196] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal/wals/b26099f8d7cd4622bb0c4637d357bf07
I20260620 22:54:59.959008 18495 ts_tablet_manager.cc:1939] T b26099f8d7cd4622bb0c4637d357bf07 P e2366f0a3b3f4e7e90386c590e2a3989: Deleting consensus metadata
I20260620 22:54:59.959410 18282 catalog_manager.cc:4841] TS e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831): tablet b26099f8d7cd4622bb0c4637d357bf07 (table test-table [id=033e76031ffb42dea3b8b130a6382248]) successfully deleted
W20260620 22:55:00.019184 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:00.019387 18491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:00.019409 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:00.020539 18491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:00.020576 18491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:00.020605 18491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:00.022002 18491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:37199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=44495
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:00.022207 18491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:00.022405 18491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:00.026895 18500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:00.026889 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:00.026917 18503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:00.027132 18491 server_base.cc:1034] running on GCE node
I20260620 22:55:00.027302 18491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:00.027518 18491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:00.028644 18491 hybrid_clock.cc:648] HybridClock initialized: now 1781996100028629 us; error 31 us; skew 500 ppm
Jun 20 22:55:00 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:00.030865 18491 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:55:00.031260 18491 webserver.cc:460] Webserver started at http://127.2.90.130:44495/ using document root <none> and password file <none>
I20260620 22:55:00.031452 18491 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:00.031512 18491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:00.032763 18491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:55:00.033246 18510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.033394 18491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.033471 18491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "72683c48431b4a65a329d2b47f47a180"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "1013835c5a876a97cd2a1359077355e0"
server_key_iv: "543d4b284c9de54f7b89951c1565db6f"
server_key_version: "encryptionkey@0"
I20260620 22:55:00.033782 18491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:00.053745 18491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:00.055467 18491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:00.055596 18491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:00.055807 18491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:00.056087 18491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:00.056119 18491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.056149 18491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:00.056169 18491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.061486 18491 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:37199
I20260620 22:55:00.061539 18623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:37199 every 8 connection(s)
I20260620 22:55:00.061838 18491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:00.064723 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18491
I20260620 22:55:00.064998 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:42747
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:35411
--webserver_port=34213
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 22:55:00 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:00.070612 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.062446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:34125 (local address 127.2.90.190:35411)
0620 22:55:00.062498 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.062500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.062681 (+ 181us) server_negotiation.cc:407] Connection header received
0620 22:55:00.063179 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.063181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.063195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.063238 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:00.064130 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.064624 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.065670 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.065881 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.065937 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.068014 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:00.068015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:00.068016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:00.068022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:00.069286 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.069733 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.069734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.069734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.069779 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.070024 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.070025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.070025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.070119 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:00.070163 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.070479 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.070547 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:55:00.070863 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.062434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:34125)
0620 22:55:00.062573 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.062606 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:00.063088 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:00.063315 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:00.063319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:00.063847 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:00.064040 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.064046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.064717 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.064718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.065591 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.065594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.065955 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.065956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.066068 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.066510 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:00.066518 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:00.067921 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:00.069358 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:00.069371 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:00.069373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:00.069664 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:00.069831 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:00.069831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:00.069832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:00.069968 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:00.070171 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:00.070176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:00.070341 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 22:55:00.070470 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 22:55:00.070637 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260620 22:55:00.071368 18624 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35411
I20260620 22:55:00.071467 18624 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:00.071642 18624 heartbeater.cc:511] Master 127.2.90.190:35411 requested a full tablet report, sending...
I20260620 22:55:00.072017 18298 ts_manager.cc:194] Registered new tserver with Master: 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
I20260620 22:55:00.072382 18298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:34125
W20260620 22:55:00.142796 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:00.142988 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:00.143009 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:00.144068 18627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:00.144098 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:00.144122 18627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:00.145460 18627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:42747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=34213
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:00.145622 18627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:00.145784 18627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:00.150843 18633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:00.150847 18632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:00.150892 18635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:00.151173 18627 server_base.cc:1034] running on GCE node
I20260620 22:55:00.151384 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:00.151636 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:00.152786 18627 hybrid_clock.cc:648] HybridClock initialized: now 1781996100152767 us; error 30 us; skew 500 ppm
Jun 20 22:55:00 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:00.155431 18627 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:55:00.155826 18627 webserver.cc:460] Webserver started at http://127.2.90.131:34213/ using document root <none> and password file <none>
I20260620 22:55:00.156016 18627 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:00.156072 18627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:00.157334 18627 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 22:55:00.157824 18642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.157943 18627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.158013 18627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d7637a8f607644cbafc8d0e84351679a"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "a0779ef7922051b7dcdac835398b8fd9"
server_key_iv: "21f60a24425859d321341017c12c707f"
server_key_version: "encryptionkey@0"
I20260620 22:55:00.158332 18627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:00.169049 18627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:00.171061 18627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:00.171182 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:00.171396 18627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:00.171671 18627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:00.171703 18627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.171722 18627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:00.171741 18627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.177043 18627 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42747
I20260620 22:55:00.177100 18755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42747 every 8 connection(s)
I20260620 22:55:00.177390 18627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:00.179142 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18627
Jun 20 22:55:00 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996100, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:00.185910 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.179496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.2.90.190:35411)
0620 22:55:00.179670 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.179672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.179723 (+ 51us) server_negotiation.cc:407] Connection header received
0620 22:55:00.179773 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.179774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.179783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.179862 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:00.180458 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.180986 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.181966 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.182178 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.182220 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.183410 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:00.183411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:00.183412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:00.183418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:00.184898 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.185216 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.185217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.185217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.185265 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.185429 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.185430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.185430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.185543 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:00.185670 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.185731 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.185801 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260620 22:55:00.186887 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.178038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:36945 (local address 127.2.90.190:35411)
0620 22:55:00.178096 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.178097 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.178281 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:55:00.178785 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.178786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.178798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.178854 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:00.180027 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.180541 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.181796 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.181997 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.182056 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.184201 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:00.184202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:00.184203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:00.184209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:00.185603 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.186043 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.186044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.186044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.186093 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.186319 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.186320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.186321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.186449 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:00.186512 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.186766 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.186824 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:55:00.187058 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.178020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:36945)
0620 22:55:00.178173 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.178205 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:00.178698 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:00.178920 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:00.178936 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:00.179638 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:00.179934 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.179940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.180623 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.180624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.181727 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.181729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.182065 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.182066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.182161 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.182581 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:00.182590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:00.184123 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:00.185674 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:00.185680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:00.185683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:00.185954 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:00.186141 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:00.186142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:00.186142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:00.186263 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:00.186516 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:00.186522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:00.186662 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 22:55:00.186745 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:55:00.186853 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260620 22:55:00.187407 18756 heartbeater.cc:347] Connected to a master server at 127.2.90.190:35411
I20260620 22:55:00.187487 18756 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:00.187644 18756 heartbeater.cc:511] Master 127.2.90.190:35411 requested a full tablet report, sending...
I20260620 22:55:00.187933 18298 ts_manager.cc:194] Registered new tserver with Master: d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747)
I20260620 22:55:00.188243 18298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:36945
I20260620 22:55:00.197285 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:00.199498 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.197581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.2.90.190:35411)
0620 22:55:00.197612 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.197613 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.197701 (+ 88us) server_negotiation.cc:407] Connection header received
0620 22:55:00.197756 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.197757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.197786 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.197836 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:00.197912 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.198362 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.199035 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.199200 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.199240 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.199328 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:00.199364 (+ 36us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 22:55:00.199455 (+ 91us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45800: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:55:00.199538 18489 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45800: token expired
W20260620 22:55:00.199644 18229 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.90.190:35411)
I20260620 22:55:00.204331 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.199860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.2.90.190:35411)
0620 22:55:00.199889 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.199891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.199899 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:00.199920 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.199920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.199927 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.199964 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:00.200320 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.200767 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.201353 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.201513 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.201548 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.202395 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:00.202396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:00.202396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:00.202401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:00.203535 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.203791 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.203791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.203791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.203832 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:00.203972 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:00.203972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:00.203972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:00.204085 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:00.204155 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.204231 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.204279 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 22:55:00.205324 18298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:00.206877 18298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:00.208910 18420 tablet_service.cc:1467] Processing CreateTablet for tablet 376263e44c9b4100b4b55715d303cc04 (DEFAULT_TABLE table=test-table [id=8b690a2a8cd742b2bfb4ad193057e160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:00.209095 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376263e44c9b4100b4b55715d303cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:00.210351 18378 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap starting.
I20260620 22:55:00.210664 18378 tablet_bootstrap.cc:654] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:00.211213 18378 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: No bootstrap required, opened a new log
I20260620 22:55:00.211267 18378 ts_tablet_manager.cc:1397] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 22:55:00.211364 18378 raft_consensus.cc:348] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.211437 18378 raft_consensus.cc:374] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:00.211483 18378 raft_consensus.cc:729] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2366f0a3b3f4e7e90386c590e2a3989, State: Initialized, Role: FOLLOWER
I20260620 22:55:00.211519 18378 consensus_queue.cc:260] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.211591 18378 ts_tablet_manager.cc:1428] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:00.211712 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.208458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37199 (local address 127.0.0.1:33378)
0620 22:55:00.208503 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.208513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:00.208557 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:00.208890 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:00.208891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:00.208893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:00.208931 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.208933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.209869 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.209870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.210683 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.210685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.211514 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.211515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.211580 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.211585 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:55:00.211612 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 22:55:00.211651 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 22:55:00.212023 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.208550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.2.90.130:37199)
0620 22:55:00.208731 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.208736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.208746 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:55:00.208785 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.208787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.208803 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.208873 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:00.208977 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.209804 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.210763 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.211425 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.211807 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.211878 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.211904 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.211944 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260620 22:55:00.212723 18558 tablet_service.cc:1467] Processing CreateTablet for tablet 376263e44c9b4100b4b55715d303cc04 (DEFAULT_TABLE table=test-table [id=8b690a2a8cd742b2bfb4ad193057e160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:00.213052 18558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376263e44c9b4100b4b55715d303cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:00.213693 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.208635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.2.90.131:42747)
0620 22:55:00.210420 (+ 1785us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.210425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.210436 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:55:00.210473 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.210476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.210494 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.210577 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:00.210661 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.211619 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.212882 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.213400 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.213456 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.213492 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.213611 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.213637 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1747,"thread_start_us":108,"threads_started":1}
I20260620 22:55:00.213712 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.208458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42747 (local address 127.0.0.1:34442)
0620 22:55:00.208737 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.208747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:00.208793 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:00.210574 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:00.210575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:00.210577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:00.210609 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.210610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.211710 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.211711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.212799 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.212802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.213476 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.213477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.213535 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.213539 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:00.213602 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:55:00.213640 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260620 22:55:00.214262 18690 tablet_service.cc:1467] Processing CreateTablet for tablet 376263e44c9b4100b4b55715d303cc04 (DEFAULT_TABLE table=test-table [id=8b690a2a8cd742b2bfb4ad193057e160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:00.214529 18690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376263e44c9b4100b4b55715d303cc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:00.215764 18766 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Bootstrap starting.
I20260620 22:55:00.216332 18766 tablet_bootstrap.cc:654] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:00.216825 18766 log.cc:824] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:00.217455 18766 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: No bootstrap required, opened a new log
I20260620 22:55:00.217509 18766 ts_tablet_manager.cc:1397] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:55:00.218847 18766 raft_consensus.cc:348] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.218994 18766 raft_consensus.cc:374] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:00.219017 18766 raft_consensus.cc:729] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72683c48431b4a65a329d2b47f47a180, State: Initialized, Role: FOLLOWER
I20260620 22:55:00.219100 18766 consensus_queue.cc:260] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.219314 18766 ts_tablet_manager.cc:1428] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:55:00.219350 18624 heartbeater.cc:503] Master 127.2.90.190:35411 was elected leader, sending a full tablet report...
I20260620 22:55:00.217183 18768 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Bootstrap starting.
I20260620 22:55:00.220026 18768 tablet_bootstrap.cc:654] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:00.220521 18768 log.cc:824] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:00.221274 18768 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: No bootstrap required, opened a new log
I20260620 22:55:00.221336 18768 ts_tablet_manager.cc:1397] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 22:55:00.222595 18768 raft_consensus.cc:348] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.222720 18768 raft_consensus.cc:374] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:00.222750 18768 raft_consensus.cc:729] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7637a8f607644cbafc8d0e84351679a, State: Initialized, Role: FOLLOWER
I20260620 22:55:00.222856 18768 consensus_queue.cc:260] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.223083 18768 ts_tablet_manager.cc:1428] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:55:00.223095 18756 heartbeater.cc:503] Master 127.2.90.190:35411 was elected leader, sending a full tablet report...
W20260620 22:55:00.312398 18625 tablet.cc:2369] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:55:00.427877 18757 tablet.cc:2369] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:55:00.434857 18487 tablet.cc:2369] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:00.454280 18492 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:00.454437 18492 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.454758 18492 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
I20260620 22:55:00.457486 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.454972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:38095 (local address 127.2.90.130:37199)
0620 22:55:00.455035 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.455037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.455127 (+ 90us) server_negotiation.cc:407] Connection header received
0620 22:55:00.455199 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.455200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.455208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.455256 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:00.455411 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.455855 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.456809 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.457177 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.457239 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.457247 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.457380 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.457418 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:55:00.457491 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.454977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:47447 (local address 127.2.90.131:42747)
0620 22:55:00.455021 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.455023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.455126 (+ 103us) server_negotiation.cc:407] Connection header received
0620 22:55:00.455185 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.455186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.455195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.455243 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:00.455387 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.455933 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.456815 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.457178 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.457238 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.457246 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.457387 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.457420 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:55:00.457510 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.454922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37199 (local address 127.2.90.129:38095)
0620 22:55:00.455061 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.455069 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:00.455134 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:00.455264 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:00.455265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:00.455267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:00.455347 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.455349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.455936 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.455937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.456730 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.456733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.457246 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.457247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.457317 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.457321 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:00.457364 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:55:00.457402 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":14,"thread_start_us":60,"threads_started":1}
I20260620 22:55:00.457530 18773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.454914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42747 (local address 127.2.90.129:47447)
0620 22:55:00.455057 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.455067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:00.455129 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:00.455259 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:00.455260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:00.455261 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:00.455332 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.455333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.456009 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.456009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.456733 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:00.456735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.457247 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.457249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:00.457319 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.457326 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 22:55:00.457381 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:55:00.457419 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260620 22:55:00.457826 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376263e44c9b4100b4b55715d303cc04" candidate_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72683c48431b4a65a329d2b47f47a180" is_pre_election: true
I20260620 22:55:00.457887 18710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376263e44c9b4100b4b55715d303cc04" candidate_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7637a8f607644cbafc8d0e84351679a" is_pre_election: true
I20260620 22:55:00.457980 18578 raft_consensus.cc:2454] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2366f0a3b3f4e7e90386c590e2a3989 in term 0.
I20260620 22:55:00.458011 18710 raft_consensus.cc:2454] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2366f0a3b3f4e7e90386c590e2a3989 in term 0.
I20260620 22:55:00.458426 18373 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72683c48431b4a65a329d2b47f47a180, e2366f0a3b3f4e7e90386c590e2a3989; no voters:
I20260620 22:55:00.458541 18492 raft_consensus.cc:2781] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:00.458606 18492 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:00.458627 18492 raft_consensus.cc:3037] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:00.459252 18492 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.459383 18492 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 1 election: Requested vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
I20260620 22:55:00.459527 18710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376263e44c9b4100b4b55715d303cc04" candidate_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7637a8f607644cbafc8d0e84351679a"
I20260620 22:55:00.459595 18710 raft_consensus.cc:3037] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:00.460230 18710 raft_consensus.cc:2454] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2366f0a3b3f4e7e90386c590e2a3989 in term 1.
I20260620 22:55:00.460347 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376263e44c9b4100b4b55715d303cc04" candidate_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72683c48431b4a65a329d2b47f47a180"
I20260620 22:55:00.460381 18372 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7637a8f607644cbafc8d0e84351679a, e2366f0a3b3f4e7e90386c590e2a3989; no voters:
I20260620 22:55:00.460419 18578 raft_consensus.cc:3037] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:00.460515 18492 raft_consensus.cc:2781] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:00.460620 18492 raft_consensus.cc:686] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 LEADER]: Becoming Leader. State: Replica: e2366f0a3b3f4e7e90386c590e2a3989, State: Running, Role: LEADER
I20260620 22:55:00.460704 18492 consensus_queue.cc:237] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:00.461278 18578 raft_consensus.cc:2454] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2366f0a3b3f4e7e90386c590e2a3989 in term 1.
I20260620 22:55:00.461499 18298 catalog_manager.cc:5495] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129). New cstate: current_term: 1 leader_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } health_report { overall_health: HEALTHY } } }
I20260620 22:55:00.493074 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.490964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.2.90.129:46831)
0620 22:55:00.491092 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:00.491094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:00.491100 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:55:00.491136 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:00.491137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:00.491154 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:00.491222 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:00.491308 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.491788 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.492433 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:00.492617 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:00.492663 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:00.492804 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:00.492910 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:00.492963 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:00.492983 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 22:55:00.493001 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 22:55:00.494489 18710 raft_consensus.cc:1261] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Refusing update from remote peer e2366f0a3b3f4e7e90386c590e2a3989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:00.494495 18578 raft_consensus.cc:1261] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Refusing update from remote peer e2366f0a3b3f4e7e90386c590e2a3989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:00.494796 18492 consensus_queue.cc:1035] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:00.494942 18774 consensus_queue.cc:1035] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:00.497303 18778 mvcc.cc:205] Tried to move back new op lower bound from 7299056027623239680 to 7299056027487633408. Current Snapshot: MvccSnapshot[applied={T|T < 7299056027623239680}]
I20260620 22:55:00.497733 18779 mvcc.cc:205] Tried to move back new op lower bound from 7299056027623239680 to 7299056027487633408. Current Snapshot: MvccSnapshot[applied={T|T < 7299056027623239680}]
I20260620 22:55:00.498310 18780 mvcc.cc:205] Tried to move back new op lower bound from 7299056027623239680 to 7299056027487633408. Current Snapshot: MvccSnapshot[applied={T|T < 7299056027623239680}]
I20260620 22:55:00.499212 18297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:00.499305 18297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:00.499963 18297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Sending DeleteTablet for 3 replicas of tablet 376263e44c9b4100b4b55715d303cc04
I20260620 22:55:00.500250 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 18336
I20260620 22:55:00.500224 18558 tablet_service.cc:1514] Processing DeleteTablet for tablet 376263e44c9b4100b4b55715d303cc04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:00 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33378
I20260620 22:55:00.500239 18690 tablet_service.cc:1514] Processing DeleteTablet for tablet 376263e44c9b4100b4b55715d303cc04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:00 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:34442
I20260620 22:55:00.500962 18788 tablet_replica.cc:330] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: stopping tablet replica
I20260620 22:55:00.501056 18788 raft_consensus.cc:2229] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:00.501148 18788 raft_consensus.cc:2258] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:00.501511 18788 ts_tablet_manager.cc:1905] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:00.503086 18788 ts_tablet_manager.cc:1918] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:00.503155 18788 log.cc:1196] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/376263e44c9b4100b4b55715d303cc04
I20260620 22:55:00.503413 18788 ts_tablet_manager.cc:1939] T 376263e44c9b4100b4b55715d303cc04 P 72683c48431b4a65a329d2b47f47a180: Deleting consensus metadata
I20260620 22:55:00.503798 18283 catalog_manager.cc:4841] TS 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): tablet 376263e44c9b4100b4b55715d303cc04 (table test-table [id=8b690a2a8cd742b2bfb4ad193057e160]) successfully deleted
I20260620 22:55:00.504805 18789 tablet_replica.cc:330] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: stopping tablet replica
I20260620 22:55:00.504905 18789 raft_consensus.cc:2229] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:00.505002 18789 raft_consensus.cc:2258] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:00.505331 18789 ts_tablet_manager.cc:1905] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:55:00.506242 18282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:46831 (error 108)
I20260620 22:55:00.506471 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 18491
I20260620 22:55:00.507133 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:00.506591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46831 (local address 127.0.0.1:59900)
0620 22:55:00.507055 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:00.507099 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
I20260620 22:55:00.507334 18789 ts_tablet_manager.cc:1918] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:00.507398 18789 log.cc:1196] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/376263e44c9b4100b4b55715d303cc04
I20260620 22:55:00.507647 18789 ts_tablet_manager.cc:1939] T 376263e44c9b4100b4b55715d303cc04 P d7637a8f607644cbafc8d0e84351679a: Deleting consensus metadata
W20260620 22:55:00.507278 18282 catalog_manager.cc:4584] TS e2366f0a3b3f4e7e90386c590e2a3989 (127.2.90.129:46831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 376263e44c9b4100b4b55715d303cc04: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46831: connect: Connection refused (error 111)
I20260620 22:55:00.508030 18282 catalog_manager.cc:4841] TS d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): tablet 376263e44c9b4100b4b55715d303cc04 (table test-table [id=8b690a2a8cd742b2bfb4ad193057e160]) successfully deleted
I20260620 22:55:00.511636 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 18627
I20260620 22:55:00.515381 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 18265
I20260620 22:55:20.519872 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35411
--webserver_interface=127.2.90.190
--webserver_port=46827
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:20.597443 18791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:20.597632 18791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:20.597653 18791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:20.598780 18791 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 22:55:20.598825 18791 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 22:55:20.598837 18791 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 22:55:20.598847 18791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:20.598858 18791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:20.598873 18791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:20.598898 18791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:20.598922 18791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:20.600355 18791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:20.600567 18791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:20.600811 18791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:20.605581 18796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:20.605610 18797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:20.605741 18799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:20.605583 18791 server_base.cc:1034] running on GCE node
I20260620 22:55:20.606011 18791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:20.606289 18791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:20.607429 18791 hybrid_clock.cc:648] HybridClock initialized: now 1781996120607417 us; error 27 us; skew 500 ppm
Jun 20 22:55:20 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996120, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:20.615650 18791 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:55:20.616196 18791 webserver.cc:460] Webserver started at http://127.2.90.190:46827/ using document root <none> and password file <none>
I20260620 22:55:20.616448 18791 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:20.616503 18791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:20.617811 18791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:20.618431 18806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:20.618573 18791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:20.618660 18791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "06c3c0ceb0d8fb8faa79b3bb8dd090cb"
server_key_iv: "fcf731a240f8cca04401396be40cc7a5"
server_key_version: "encryptionkey@0"
I20260620 22:55:20.618999 18791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:20.632624 18791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:20.634459 18791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:20.634582 18791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:20.638230 18791 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:35411
I20260620 22:55:20.638266 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:35411 every 8 connection(s)
I20260620 22:55:20.638599 18791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:20.639958 18859 sys_catalog.cc:263] Verifying existing consensus state
I20260620 22:55:20.640581 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap starting.
I20260620 22:55:20.644388 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18791
I20260620 22:55:20.644668 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:46831
--local_ip_for_outbound_sockets=127.2.90.129
--tserver_master_addrs=127.2.90.190:35411
--webserver_port=32805
--webserver_interface=127.2.90.129
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:55:20.648607 18859 log.cc:824] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:20.650655 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:55:20.650918 18859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Bootstrap complete.
I20260620 22:55:20.652944 18859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:55:20.653156 18859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b0eb5c83a854b7dabdab9ef6793f5d2, State: Initialized, Role: FOLLOWER
I20260620 22:55:20.653304 18859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:55:20.653379 18859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:20.653426 18859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:20.653471 18859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 22:55:20.654379 18859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:55:20.654469 18859 leader_election.cc:302] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b0eb5c83a854b7dabdab9ef6793f5d2; no voters:
I20260620 22:55:20.654609 18859 leader_election.cc:288] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 22:55:20.654657 18863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 22:55:20.654857 18859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:20.654860 18863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [term 3 LEADER]: Becoming Leader. State: Replica: 0b0eb5c83a854b7dabdab9ef6793f5d2, State: Running, Role: LEADER
I20260620 22:55:20.655045 18863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } }
I20260620 22:55:20.655349 18864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } } }
I20260620 22:55:20.655412 18864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:20.655599 18865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b0eb5c83a854b7dabdab9ef6793f5d2. Latest consensus state: current_term: 3 leader_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b0eb5c83a854b7dabdab9ef6793f5d2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 35411 } } }
I20260620 22:55:20.655690 18865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:20.655715 18871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:20.656466 18871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:20.656563 18871 catalog_manager.cc:1259] Loaded cluster ID: f4a58789ffc64b45928732feeaf8bea2
I20260620 22:55:20.656634 18871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:20.657541 18871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:20.657727 18871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Loaded TSK: 0
I20260620 22:55:20.657766 18871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b0eb5c83a854b7dabdab9ef6793f5d2: Loaded TSK: 1
F20260620 22:55:20.657966 18871 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f53ab7049ad google::LogMessage::Fail() at ??:0
@ 0x7f53ab70788a google::LogMessage::SendToLog() at ??:0
@ 0x7f53ab7044de google::LogMessage::Flush() at ??:0
@ 0x7f53ab705a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f53ade71104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f53ade6ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f53b365e1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f53b365e48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f53b365e594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f53b3624c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f53b362ead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f53ad5d20d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f53ad5c7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f53ae42b6db start_thread at ??:0
@ 0x7f53aa20261f clone at ??:0
W20260620 22:55:20.739327 18861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:20.739508 18861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:20.739529 18861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:20.740621 18861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:20.740657 18861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:20.740712 18861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:20.742094 18861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:46831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=32805
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:20.742300 18861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:20.742509 18861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:20.747061 18900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:20.747071 18901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:20.747255 18861 server_base.cc:1034] running on GCE node
W20260620 22:55:20.747071 18903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:20.747539 18861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:20.747747 18861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:20.748883 18861 hybrid_clock.cc:648] HybridClock initialized: now 1781996120748871 us; error 30 us; skew 500 ppm
Jun 20 22:55:20 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996120, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:20.751062 18861 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:55:20.751444 18861 webserver.cc:460] Webserver started at http://127.2.90.129:32805/ using document root <none> and password file <none>
I20260620 22:55:20.751626 18861 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:20.751672 18861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:20.752964 18861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:20.753554 18910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:20.753695 18861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:20.753785 18861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "e2366f0a3b3f4e7e90386c590e2a3989"
format_stamp: "Formatted at 2026-06-20 22:54:38 on dist-test-slave-4mtw"
server_key: "858700cf71434e38e50a4cb6800d97cb"
server_key_iv: "30daaf2943c7d0a382a8fc6103185c44"
server_key_version: "encryptionkey@0"
I20260620 22:55:20.754081 18861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:20.797147 18861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:20.799263 18861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:20.799402 18861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:20.799628 18861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:20.800056 18917 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 22:55:20.801982 18861 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 22:55:20.802035 18861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 22:55:20.802070 18861 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 22:55:20.802677 18861 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 22:55:20.802712 18861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 22:55:20.802815 18917 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap starting.
I20260620 22:55:20.809194 18861 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:46831
I20260620 22:55:20.809551 18861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:55:20.809727 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 18861
I20260620 22:55:20.810042 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:37199
--local_ip_for_outbound_sockets=127.2.90.130
--tserver_master_addrs=127.2.90.190:35411
--webserver_port=44495
--webserver_interface=127.2.90.130
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:55:20.809854 19024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:46831 every 8 connection(s)
I20260620 22:55:20.815208 18917 log.cc:824] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:20.818171 18917 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 22:55:20.818482 18917 tablet_bootstrap.cc:492] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Bootstrap complete.
I20260620 22:55:20.819528 18917 ts_tablet_manager.cc:1397] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260620 22:55:20.820947 18917 raft_consensus.cc:348] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:20.821277 18917 raft_consensus.cc:729] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2366f0a3b3f4e7e90386c590e2a3989, State: Initialized, Role: FOLLOWER
I20260620 22:55:20.821448 18917 consensus_queue.cc:260] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:20.821772 18917 ts_tablet_manager.cc:1428] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260620 22:55:20.834108 19027 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:55:20.810242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:57029)
0620 22:55:20.810425 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:20.810466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:20.810892 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:20.833923 (+ 23031us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
W20260620 22:55:20.834345 19025 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35411 (0 consecutive failures): Network error: Failed to ping master at 127.2.90.190:35411: Client connection negotiation failed: client connection to 127.2.90.190:35411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:55:20.834635 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:20.834536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:60485)
0620 22:55:20.834581 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:20.834614 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:55:20.834928 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:20.834862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:33893)
0620 22:55:20.834893 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:20.834913 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 22:55:20.835031 19025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 22:55:20.890216 19028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:20.890409 19028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:20.890431 19028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:20.891541 19028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:20.891583 19028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:20.891611 19028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:20.893085 19028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:37199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=44495
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:20.893290 19028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:20.893484 19028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:20.897976 19038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:20.897994 19036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:20.898115 19028 server_base.cc:1034] running on GCE node
W20260620 22:55:20.898005 19035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:20.898370 19028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:20.898579 19028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:20.899719 19028 hybrid_clock.cc:648] HybridClock initialized: now 1781996120899700 us; error 29 us; skew 500 ppm
Jun 20 22:55:20 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996120, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:20.902069 19028 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:55:20.902470 19028 webserver.cc:460] Webserver started at http://127.2.90.130:44495/ using document root <none> and password file <none>
I20260620 22:55:20.902647 19028 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:20.902684 19028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:20.903918 19028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:55:20.904548 19045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:20.904728 19028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:55:20.904796 19028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "72683c48431b4a65a329d2b47f47a180"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "1013835c5a876a97cd2a1359077355e0"
server_key_iv: "543d4b284c9de54f7b89951c1565db6f"
server_key_version: "encryptionkey@0"
I20260620 22:55:20.905109 19028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:20.918517 19028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:20.920145 19028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:20.920253 19028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:20.920446 19028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:20.920751 19028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:20.920787 19028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:20.920815 19028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:20.920847 19028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:20.926040 19028 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:37199
I20260620 22:55:20.926105 19158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:37199 every 8 connection(s)
I20260620 22:55:20.926406 19028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:20.927418 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:20.927021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:49119)
0620 22:55:20.927196 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:20.927268 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
W20260620 22:55:20.927786 19159 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35411 (0 consecutive failures): Network error: Failed to ping master at 127.2.90.190:35411: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
I20260620 22:55:20.928028 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:20.927948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:45127)
0620 22:55:20.927985 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:20.928010 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 22:55:20.928227 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:20.928171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:43845)
0620 22:55:20.928198 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:20.928215 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 22:55:20.928313 19159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:55:20.934981 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19028
I20260620 22:55:20.935216 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:42747
--local_ip_for_outbound_sockets=127.2.90.131
--tserver_master_addrs=127.2.90.190:35411
--webserver_port=34213
--webserver_interface=127.2.90.131
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:21.014451 19162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:21.014647 19162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:21.014668 19162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:21.015741 19162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:21.015774 19162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:21.015805 19162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:21.017189 19162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:42747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=34213
--tserver_master_addrs=127.2.90.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:21.017362 19162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:21.017529 19162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:21.022132 19167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:21.022122 19168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:21.022109 19170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:21.022324 19162 server_base.cc:1034] running on GCE node
I20260620 22:55:21.022505 19162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:21.022710 19162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:21.023859 19162 hybrid_clock.cc:648] HybridClock initialized: now 1781996121023840 us; error 34 us; skew 500 ppm
Jun 20 22:55:21 dist-test-slave-4mtw krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:21.026173 19162 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:55:21.026584 19162 webserver.cc:460] Webserver started at http://127.2.90.131:34213/ using document root <none> and password file <none>
I20260620 22:55:21.026764 19162 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:21.026806 19162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:21.027979 19162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:21.028463 19177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:21.028602 19162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:21.028673 19162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "d7637a8f607644cbafc8d0e84351679a"
format_stamp: "Formatted at 2026-06-20 22:54:39 on dist-test-slave-4mtw"
server_key: "a0779ef7922051b7dcdac835398b8fd9"
server_key_iv: "21f60a24425859d321341017c12c707f"
server_key_version: "encryptionkey@0"
I20260620 22:55:21.029018 19162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:21.039871 19162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:21.041816 19162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:21.041942 19162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:21.042132 19162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:21.042395 19162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:21.042428 19162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:21.042454 19162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:21.042476 19162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:21.047897 19162 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:42747
I20260620 22:55:21.047935 19290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:42747 every 8 connection(s)
I20260620 22:55:21.048256 19162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:21.049264 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:21.048899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:38077)
0620 22:55:21.049048 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:21.049119 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
W20260620 22:55:21.049641 19291 heartbeater.cc:644] Failed to heartbeat to 127.2.90.190:35411 (0 consecutive failures): Network error: Failed to ping master at 127.2.90.190:35411: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
I20260620 22:55:21.049742 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19162
I20260620 22:55:21.049947 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:21.049863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:60067)
0620 22:55:21.049895 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:21.049926 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 22:55:21.050239 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:21.050160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:50365)
0620 22:55:21.050196 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:21.050218 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:55:21.050354 19291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 22:55:21.835659 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:21.835427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:60025)
0620 22:55:21.835558 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:21.835598 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260620 22:55:21.928786 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:21.928581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:41915)
0620 22:55:21.928711 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:21.928742 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260620 22:55:22.050827 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.050644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:36537)
0620 22:55:22.050745 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:22.050777 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260620 22:55:22.295923 19298 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:22.296056 19298 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:22.296368 19298 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
Jun 20 22:55:22 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996120, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.130@KRBTEST.COM
Jun 20 22:55:22 dist-test-slave-4mtw krb5kdc[17721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996120, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.131@KRBTEST.COM
I20260620 22:55:22.306137 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.296621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:37569 (local address 127.2.90.131:42747)
0620 22:55:22.296776 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:22.296789 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:22.296819 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:55:22.297325 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:22.297328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:22.297367 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:22.297491 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:22.298235 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.299064 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.300065 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.300451 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.300544 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:22.303156 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:22.303160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:22.303160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:22.303187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:22.304560 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:22.305047 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:22.305050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:22.305050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:22.305135 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:22.305379 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:22.305380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:22.305381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:22.305592 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:22.305657 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:22.305914 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 22:55:22.306031 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260620 22:55:22.306137 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.296657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:50427 (local address 127.2.90.130:37199)
0620 22:55:22.296809 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:22.296820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:22.296850 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:55:22.297303 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:22.297305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:22.297350 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:22.297475 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:22.298212 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.299050 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.300063 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.300451 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.300544 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:22.302772 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:22.302776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:22.302776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:22.302788 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:22.304577 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:22.305046 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:22.305049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:22.305049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:22.305136 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:22.305365 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:22.305366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:22.305366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:22.305581 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:22.305647 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:22.305911 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 22:55:22.306032 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260620 22:55:22.306142 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.296513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:42747 (local address 127.2.90.129:37569)
0620 22:55:22.296731 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:22.296740 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:22.296812 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:22.297518 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:22.297521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:22.297908 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:22.298100 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:22.298106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.299159 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.299160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:22.299986 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:22.299989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.300528 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.300529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:22.300633 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:22.301186 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:22.301196 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:22.303052 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:22.304631 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:22.304637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:22.304639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:22.304976 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:22.305192 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:22.305193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:22.305193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:22.305315 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:22.305651 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:22.305652 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:22.305762 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 22:55:22.305838 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:55:22.306012 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":114,"thread_start_us":83,"threads_started":1}
I20260620 22:55:22.306558 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.296513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:37199 (local address 127.2.90.129:50427)
0620 22:55:22.296569 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:22.296581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:22.296652 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:22.297518 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:22.297521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:22.297893 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:22.298099 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:22.298105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.299139 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.299140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:22.299985 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:22.299988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:22.300527 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:22.300528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:22.300634 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:22.301231 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:22.301236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:22.302645 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:22.304644 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:22.304645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:22.304645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:22.304976 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:22.305190 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:22.305191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:22.305191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:22.305304 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:22.305638 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:22.305644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:22.305762 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 22:55:22.305838 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:55:22.306080 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":154}
W20260620 22:55:22.307132 18911 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
W20260620 22:55:22.307204 18912 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
I20260620 22:55:22.307288 18911 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2366f0a3b3f4e7e90386c590e2a3989; no voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a
I20260620 22:55:22.307399 19298 raft_consensus.cc:2726] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:55:22.836370 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.836130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:53081)
0620 22:55:22.836283 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:22.836321 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260620 22:55:22.929342 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:22.929157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:38599)
0620 22:55:22.929268 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:22.929303 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260620 22:55:23.051399 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:23.051188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:49169)
0620 22:55:23.051322 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:23.051358 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260620 22:55:23.836997 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:23.836825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:54151)
0620 22:55:23.836920 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:23.836950 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
I20260620 22:55:23.929920 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:23.929704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:48357)
0620 22:55:23.929812 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:23.929880 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260620 22:55:24.052011 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:24.051814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:48875)
0620 22:55:24.051933 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:24.051970 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260620 22:55:24.142480 19308 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:24.142566 19308 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:24.142705 19308 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
W20260620 22:55:24.143057 18912 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
W20260620 22:55:24.143141 18911 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
I20260620 22:55:24.143182 18911 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2366f0a3b3f4e7e90386c590e2a3989; no voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a
I20260620 22:55:24.143253 19308 raft_consensus.cc:2726] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:55:24.837551 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:24.837371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:60121)
0620 22:55:24.837479 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:24.837507 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260620 22:55:24.930477 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:24.930289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:59587)
0620 22:55:24.930394 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:24.930426 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260620 22:55:25.052603 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:25.052408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:56497)
0620 22:55:25.052529 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:25.052564 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 22:55:25.838124 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:25.837942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:56951)
0620 22:55:25.838053 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:25.838082 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260620 22:55:25.931042 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:25.930850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:55859)
0620 22:55:25.930966 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:25.931000 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260620 22:55:25.956734 19314 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:25.956815 19314 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:25.956949 19314 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
W20260620 22:55:25.957259 18911 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
W20260620 22:55:25.957356 18912 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
I20260620 22:55:25.957424 18912 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2366f0a3b3f4e7e90386c590e2a3989; no voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a
I20260620 22:55:25.957499 19314 raft_consensus.cc:2726] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:55:26.053181 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:26.053002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:50357)
0620 22:55:26.053112 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:26.053143 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260620 22:55:26.838730 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:26.838531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:35065)
0620 22:55:26.838648 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:26.838685 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 22:55:26.931600 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:26.931416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:39155)
0620 22:55:26.931523 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:26.931557 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":73,"threads_started":1}
I20260620 22:55:27.053802 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:27.053581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:36897)
0620 22:55:27.053723 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:27.053760 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260620 22:55:27.839357 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:27.839132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:48415)
0620 22:55:27.839275 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:27.839310 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260620 22:55:27.932159 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:27.931950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:39883)
0620 22:55:27.932084 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:27.932119 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260620 22:55:28.054421 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:28.054197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:38301)
0620 22:55:28.054342 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:28.054381 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 22:55:28.840068 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:28.839812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:50937)
0620 22:55:28.839989 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:28.840023 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260620 22:55:28.841636 19323 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:28.841707 19323 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:28.841869 19323 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
W20260620 22:55:28.842152 18912 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
W20260620 22:55:28.842233 18911 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
I20260620 22:55:28.842298 18911 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2366f0a3b3f4e7e90386c590e2a3989; no voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a
I20260620 22:55:28.842365 19323 raft_consensus.cc:2726] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:55:28.932737 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:28.932516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:56391)
0620 22:55:28.932637 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:28.932670 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260620 22:55:29.055032 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:29.054841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:53501)
0620 22:55:29.054958 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:29.054991 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260620 22:55:29.840649 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:29.840445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:39485)
0620 22:55:29.840572 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:29.840604 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260620 22:55:29.933279 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:29.933078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:49303)
0620 22:55:29.933202 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:29.933235 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260620 22:55:30.055624 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:30.055401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:44207)
0620 22:55:30.055544 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:30.055580 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260620 22:55:30.841255 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:30.841061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:36079)
0620 22:55:30.841174 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:30.841211 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260620 22:55:30.933820 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:30.933635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:33657)
0620 22:55:30.933741 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:30.933777 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260620 22:55:31.056236 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:31.055992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:42633)
0620 22:55:31.056149 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:31.056184 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260620 22:55:31.841866 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:31.841669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:38615)
0620 22:55:31.841786 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:31.841819 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260620 22:55:31.934464 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:31.934214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:32939)
0620 22:55:31.934384 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:31.934420 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260620 22:55:32.056901 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:32.056647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:40691)
0620 22:55:32.056802 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:32.056841 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260620 22:55:32.774593 19335 raft_consensus.cc:482] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:32.774688 19335 raft_consensus.cc:504] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7637a8f607644cbafc8d0e84351679a" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 42747 } } peers { permanent_uuid: "72683c48431b4a65a329d2b47f47a180" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 37199 } } peers { permanent_uuid: "e2366f0a3b3f4e7e90386c590e2a3989" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46831 } }
I20260620 22:55:32.774863 19335 leader_election.cc:288] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747), 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199)
W20260620 22:55:32.775218 18911 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7637a8f607644cbafc8d0e84351679a (127.2.90.131:42747): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
W20260620 22:55:32.775293 18912 leader_election.cc:341] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72683c48431b4a65a329d2b47f47a180 (127.2.90.130:37199): Not found: Tablet not found: 376263e44c9b4100b4b55715d303cc04
I20260620 22:55:32.775365 18912 leader_election.cc:302] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2366f0a3b3f4e7e90386c590e2a3989; no voters: 72683c48431b4a65a329d2b47f47a180, d7637a8f607644cbafc8d0e84351679a
I20260620 22:55:32.775439 19335 raft_consensus.cc:2726] T 376263e44c9b4100b4b55715d303cc04 P e2366f0a3b3f4e7e90386c590e2a3989 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 22:55:32.842545 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:32.842334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:58037)
0620 22:55:32.842466 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:32.842499 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260620 22:55:32.935045 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:32.934860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:32891)
0620 22:55:32.934969 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:32.935002 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260620 22:55:33.057487 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:33.057271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:46479)
0620 22:55:33.057388 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:33.057426 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 22:55:33.843214 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:33.842991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:52685)
0620 22:55:33.843133 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:33.843169 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260620 22:55:33.935642 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:33.935450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:44019)
0620 22:55:33.935567 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:33.935600 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260620 22:55:34.058100 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:34.057899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:51205)
0620 22:55:34.058021 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:34.058059 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260620 22:55:34.843791 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:34.843608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:40609)
0620 22:55:34.843715 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:34.843745 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260620 22:55:34.936206 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:34.936017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:50369)
0620 22:55:34.936134 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:34.936164 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260620 22:55:35.058624 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:35.058449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:58293)
0620 22:55:35.058552 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:35.058580 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260620 22:55:35.844491 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:35.844290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.129:36107)
0620 22:55:35.844409 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:35.844446 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260620 22:55:35.936866 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:35.936648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.130:56587)
0620 22:55:35.936789 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:35.936825 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260620 22:55:36.057993 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 18861
I20260620 22:55:36.059652 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:36.059470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:35411 (local address 127.2.90.131:37645)
0620 22:55:36.059578 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:36.059613 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.190:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260620 22:55:36.062276 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19028
I20260620 22:55:36.065953 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19162
2026-06-20T22:55:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58387 ms)
[----------] 30 tests from SecurityITest (223231 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:36 dist-test-slave-4mtw krb5kdc[19350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:36 dist-test-slave-4mtw krb5kdc[19350](info): set up 2 sockets
Jun 20 22:55:36 dist-test-slave-4mtw krb5kdc[19350](info): commencing operation
krb5kdc: starting...
W20260620 22:55:37.012104 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:55:37.116923 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41995
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:41995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:37.199298 19363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:37.199501 19363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:37.199528 19363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:37.200661 19363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:37.200732 19363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:37.200753 19363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:37.200768 19363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:37.200785 19363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:37.202205 19363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:41995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:37.202435 19363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:37.202620 19363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:37.207162 19371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:37.207203 19363 server_base.cc:1034] running on GCE node
W20260620 22:55:37.207259 19369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:37.207378 19368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:37.207616 19363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:37.207870 19363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:37.209004 19363 hybrid_clock.cc:648] HybridClock initialized: now 1781996137208989 us; error 30 us; skew 500 ppm
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:37.211154 19363 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:55:37.211547 19363 webserver.cc:460] Webserver started at http://127.2.90.190:35393/ using document root <none> and password file <none>
I20260620 22:55:37.211740 19363 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:37.211788 19363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:37.211908 19363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:37.212955 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "eab658e6ae2846679f839519d30233c2"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "b7f8a7fd7ab65068ddd2d9713e8c18c6"
server_key_iv: "9e0c8f197841dbc7970759729d5d553f"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.213263 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "eab658e6ae2846679f839519d30233c2"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "b7f8a7fd7ab65068ddd2d9713e8c18c6"
server_key_iv: "9e0c8f197841dbc7970759729d5d553f"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.214459 19363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:37.215068 19378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.215207 19363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:37.215286 19363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "eab658e6ae2846679f839519d30233c2"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "b7f8a7fd7ab65068ddd2d9713e8c18c6"
server_key_iv: "9e0c8f197841dbc7970759729d5d553f"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.215342 19363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:37.228507 19363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:37.230309 19363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:37.230455 19363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:37.234081 19363 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:41995
I20260620 22:55:37.234115 19430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:41995 every 8 connection(s)
I20260620 22:55:37.234433 19363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:37.235224 19431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:37.237426 19431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: Bootstrap starting.
I20260620 22:55:37.237964 19431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:37.238211 19431 log.cc:824] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:37.238860 19431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: No bootstrap required, opened a new log
I20260620 22:55:37.240420 19431 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab658e6ae2846679f839519d30233c2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41995 } }
I20260620 22:55:37.240557 19431 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:37.240577 19431 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab658e6ae2846679f839519d30233c2, State: Initialized, Role: FOLLOWER
I20260620 22:55:37.240733 19431 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab658e6ae2846679f839519d30233c2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41995 } }
I20260620 22:55:37.240800 19431 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:37.240826 19431 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:37.240872 19431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:37.240994 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19363
I20260620 22:55:37.241097 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:37.241281 2410 external_mini_cluster.cc:1428] Setting key 9dd28dd7509c7a42f7f8f35b14a632ec
I20260620 22:55:37.241506 19431 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab658e6ae2846679f839519d30233c2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41995 } }
I20260620 22:55:37.241600 19431 leader_election.cc:302] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab658e6ae2846679f839519d30233c2; no voters:
I20260620 22:55:37.241796 19431 leader_election.cc:288] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:37.241825 19435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:37.241964 19435 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [term 1 LEADER]: Becoming Leader. State: Replica: eab658e6ae2846679f839519d30233c2, State: Running, Role: LEADER
I20260620 22:55:37.242048 19435 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab658e6ae2846679f839519d30233c2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41995 } }
I20260620 22:55:37.242091 19431 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:37.242508 19438 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab658e6ae2846679f839519d30233c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab658e6ae2846679f839519d30233c2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41995 } } }
I20260620 22:55:37.242615 19438 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:37.242787 19437 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab658e6ae2846679f839519d30233c2. Latest consensus state: current_term: 1 leader_uuid: "eab658e6ae2846679f839519d30233c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab658e6ae2846679f839519d30233c2" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41995 } } }
I20260620 22:55:37.242852 19437 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2 [sys.catalog]: This master's current role is: LEADER
W20260620 22:55:37.243688 19452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:55:37.243760 19452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:55:37.244668 19445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:37.244853 19445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:37.246212 19445 catalog_manager.cc:1347] Generated new cluster ID: 9364089be66f417a925ef0da3798817d
I20260620 22:55:37.246304 19445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:37.253034 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.241708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.2.90.190:41995)
0620 22:55:37.242378 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.242387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.242424 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:55:37.242547 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.242554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.242610 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.243499 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:37.243595 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.244538 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.245600 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.246141 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.246244 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.248966 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:37.248969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:37.248970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:37.248982 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:37.251439 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.251821 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.251823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.251824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.251935 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.252123 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.252124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.252124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.252420 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:37.252555 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.252695 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.252824 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":92,"threads_started":1}
I20260620 22:55:37.259101 19445 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:37.259701 19445 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:37.264273 19445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: Generated new TSK 0
I20260620 22:55:37.264438 19445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:55:37.288298 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41995
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:37.366144 19457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:37.366333 19457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:37.366354 19457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:37.367452 19457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:37.367491 19457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:37.367518 19457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:37.368944 19457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:37.369153 19457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:37.369354 19457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:37.373939 19465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:37.373944 19462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:37.374037 19457 server_base.cc:1034] running on GCE node
W20260620 22:55:37.373955 19463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:37.374382 19457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:37.374595 19457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:37.375725 19457 hybrid_clock.cc:648] HybridClock initialized: now 1781996137375712 us; error 28 us; skew 500 ppm
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:37.378023 19457 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:55:37.378381 19457 webserver.cc:460] Webserver started at http://127.2.90.129:33677/ using document root <none> and password file <none>
I20260620 22:55:37.378557 19457 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:37.378595 19457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:37.378682 19457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:37.379632 19457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "5897e621532b44368e7467f8890d23ee"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "84a576fb39ceeddd6330a1fa7cb5a90d"
server_key_iv: "a73a91f95c4547bc4c6b2161f1e3662d"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.379917 19457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "5897e621532b44368e7467f8890d23ee"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "84a576fb39ceeddd6330a1fa7cb5a90d"
server_key_iv: "a73a91f95c4547bc4c6b2161f1e3662d"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.381068 19457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:37.381651 19472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.381793 19457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:37.381899 19457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "5897e621532b44368e7467f8890d23ee"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "84a576fb39ceeddd6330a1fa7cb5a90d"
server_key_iv: "a73a91f95c4547bc4c6b2161f1e3662d"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.381958 19457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:37.400728 19457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:37.402815 19457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:37.402954 19457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:37.403164 19457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:37.403442 19457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:37.403474 19457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.403502 19457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:37.403527 19457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.408902 19457 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:39367
I20260620 22:55:37.408959 19585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:39367 every 8 connection(s)
I20260620 22:55:37.409250 19457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:55:37.412406 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19457
I20260620 22:55:37.412497 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:55:37.412662 2410 external_mini_cluster.cc:1428] Setting key ae8f5cd113e4c7f7491a8bd0569f8327
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:37.418684 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.409970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:54363 (local address 127.2.90.190:41995)
0620 22:55:37.410025 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.410027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.410214 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:55:37.410711 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.410712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.410729 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.410780 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:37.411690 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.412262 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.413475 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.413731 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.413804 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.416004 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:37.416006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:37.416006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:37.416012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:37.417266 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.417770 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.417771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.417771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.417822 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.418095 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.418095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.418096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.418194 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:37.418240 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.418552 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.418619 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:55:37.418946 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.409905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41995 (local address 127.2.90.129:54363)
0620 22:55:37.410094 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.410131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.410617 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.410852 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.410855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.411400 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:37.411594 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.411600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.412341 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.412342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.413406 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.413409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.413802 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.413803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.413901 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.414349 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:37.414359 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:37.415907 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:37.417394 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:37.417402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:37.417404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:37.417703 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:37.417888 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:37.417890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:37.417890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:37.418029 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:37.418249 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:37.418254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:37.418420 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.418538 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.418701 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260620 22:55:37.419395 19586 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41995
I20260620 22:55:37.419486 19586 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:37.419656 19586 heartbeater.cc:511] Master 127.2.90.190:41995 requested a full tablet report, sending...
I20260620 22:55:37.420214 19395 ts_manager.cc:194] Registered new tserver with Master: 5897e621532b44368e7467f8890d23ee (127.2.90.129:39367)
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:55:37.421133 19395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:54363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:55:37.435374 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41995
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:37.512921 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:37.513118 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:37.513139 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:37.514240 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:37.514274 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:37.514299 19591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:37.515669 19591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:37.515839 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:37.516004 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:37.520547 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:37.520609 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:37.520525 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:37.520630 19591 server_base.cc:1034] running on GCE node
I20260620 22:55:37.520979 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:37.521188 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:37.522315 19591 hybrid_clock.cc:648] HybridClock initialized: now 1781996137522303 us; error 29 us; skew 500 ppm
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:37.527290 19591 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:55:37.527724 19591 webserver.cc:460] Webserver started at http://127.2.90.130:40041/ using document root <none> and password file <none>
I20260620 22:55:37.527917 19591 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:37.527964 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:37.528084 19591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:37.529111 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "0bbe119d9a8d43b2b7f6fb396177d30a"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "82a7f41274a04c894f0176036f706e9a"
server_key_iv: "74b2c4a7030cb5412d800e25a871cff6"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.529431 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "0bbe119d9a8d43b2b7f6fb396177d30a"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "82a7f41274a04c894f0176036f706e9a"
server_key_iv: "74b2c4a7030cb5412d800e25a871cff6"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.530576 19591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:37.531170 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.531313 19591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:37.531402 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "0bbe119d9a8d43b2b7f6fb396177d30a"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "82a7f41274a04c894f0176036f706e9a"
server_key_iv: "74b2c4a7030cb5412d800e25a871cff6"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.531458 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:37.557310 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:37.559440 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:37.559576 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:37.559788 19591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:37.560092 19591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:37.560127 19591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.560154 19591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:37.560173 19591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.565516 19591 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:40283
I20260620 22:55:37.565572 19719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:40283 every 8 connection(s)
I20260620 22:55:37.565848 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:37.569540 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19591
I20260620 22:55:37.569643 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:55:37.569840 2410 external_mini_cluster.cc:1428] Setting key a88dde385e8a66a3652b5c29455a44b0
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:37.574524 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.566484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:38743 (local address 127.2.90.190:41995)
0620 22:55:37.566550 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.566552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.566736 (+ 184us) server_negotiation.cc:407] Connection header received
0620 22:55:37.567181 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.567182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.567191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.567240 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:37.568096 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.568583 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.569623 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.569925 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.569996 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.572091 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:37.572093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:37.572093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:37.572100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:37.573353 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.573702 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.573703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.573704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.573760 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.573961 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.573962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.573962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.574070 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:37.574124 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.574360 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.574439 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:55:37.574687 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.566461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41995 (local address 127.2.90.130:38743)
0620 22:55:37.566624 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.566658 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.567106 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.567305 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.567308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.567832 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:37.568031 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.568036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.568660 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.568661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.569534 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.569536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.570006 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.570007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.570143 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.570608 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:37.570619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:37.572004 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:37.573426 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:37.573433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:37.573435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:37.573638 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:37.573812 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:37.573812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:37.573813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:37.573907 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:37.574122 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:37.574128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:37.574249 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.574331 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.574485 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 22:55:37.575220 19720 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41995
I20260620 22:55:37.575302 19720 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:37.575464 19720 heartbeater.cc:511] Master 127.2.90.190:41995 requested a full tablet report, sending...
I20260620 22:55:37.575806 19395 ts_manager.cc:194] Registered new tserver with Master: 0bbe119d9a8d43b2b7f6fb396177d30a (127.2.90.130:40283)
I20260620 22:55:37.576282 19395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:38743
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:55:37.592433 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41995
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:37.670607 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:37.670804 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:37.670833 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:37.671896 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:37.671928 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:37.671952 19725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:37.673394 19725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:37.673568 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:37.673734 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:37.678206 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:37.678300 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:37.678229 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:37.678959 19725 server_base.cc:1034] running on GCE node
I20260620 22:55:37.679132 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:37.679307 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:37.680433 19725 hybrid_clock.cc:648] HybridClock initialized: now 1781996137680424 us; error 26 us; skew 500 ppm
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:37.682677 19725 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:55:37.683048 19725 webserver.cc:460] Webserver started at http://127.2.90.131:34761/ using document root <none> and password file <none>
I20260620 22:55:37.683207 19725 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:37.683246 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:37.683332 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:37.684295 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "ddd09c0708ac46268efd1877fa3bcd38"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "225fbc8327d8eecd26f132356a044eb3"
server_key_iv: "9284c367c65e33dc9280b91206359d98"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.684589 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "ddd09c0708ac46268efd1877fa3bcd38"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "225fbc8327d8eecd26f132356a044eb3"
server_key_iv: "9284c367c65e33dc9280b91206359d98"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.685807 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:55:37.686470 19740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.686641 19725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:37.686729 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "ddd09c0708ac46268efd1877fa3bcd38"
format_stamp: "Formatted at 2026-06-20 22:55:37 on dist-test-slave-4mtw"
server_key: "225fbc8327d8eecd26f132356a044eb3"
server_key_iv: "9284c367c65e33dc9280b91206359d98"
server_key_version: "encryptionkey@0"
I20260620 22:55:37.686802 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:37.703300 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:37.705355 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:37.705485 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:37.705683 19725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:37.705967 19725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:37.706000 19725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.706028 19725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:37.706050 19725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:37.711411 19725 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:33059
I20260620 22:55:37.711475 19853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:33059 every 8 connection(s)
I20260620 22:55:37.711738 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:37.716477 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19725
I20260620 22:55:37.716575 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:55:37.716789 2410 external_mini_cluster.cc:1428] Setting key 087596a90df2c4e70cdb181f402e6499
Jun 20 22:55:37 dist-test-slave-4mtw krb5kdc[19350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:37.719940 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.712376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:39217 (local address 127.2.90.190:41995)
0620 22:55:37.712455 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.712457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.712596 (+ 139us) server_negotiation.cc:407] Connection header received
0620 22:55:37.713127 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.713128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.713139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.713194 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:37.714047 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.714532 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.715458 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.715643 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.715693 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.717798 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:37.717799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:37.717799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:37.717806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:37.718880 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.719253 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.719254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.719254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.719300 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.719475 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.719476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.719476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.719571 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:37.719615 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.719805 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.719876 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 22:55:37.720106 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.712345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:41995 (local address 127.2.90.131:39217)
0620 22:55:37.712475 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.712509 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.713021 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.713256 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.713260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.713798 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:37.713981 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.713986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.714589 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.714589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.715397 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.715399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.715700 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.715700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.715788 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.716236 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:37.716247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:37.717730 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:37.718953 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:37.718958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:37.718960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:37.719198 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:37.719343 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:37.719344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:37.719344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:37.719434 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:37.719612 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:37.719614 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:37.719719 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.719791 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.719912 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260620 22:55:37.720574 19854 heartbeater.cc:347] Connected to a master server at 127.2.90.190:41995
I20260620 22:55:37.720664 19854 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:37.720867 19854 heartbeater.cc:511] Master 127.2.90.190:41995 requested a full tablet report, sending...
I20260620 22:55:37.721196 19395 ts_manager.cc:194] Registered new tserver with Master: ddd09c0708ac46268efd1877fa3bcd38 (127.2.90.131:33059)
I20260620 22:55:37.721544 19395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:39217
I20260620 22:55:37.728413 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:37.734776 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.729710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.2.90.190:41995)
0620 22:55:37.729766 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.729767 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.729834 (+ 67us) server_negotiation.cc:407] Connection header received
0620 22:55:37.729876 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.729876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.729885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.729927 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:37.730407 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.730881 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.731552 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.731731 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.731781 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.732870 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:37.732871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:37.732871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:37.732876 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:37.733940 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.734217 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.734218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.734218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.734262 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:37.734384 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:37.734384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:37.734385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:37.734513 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:37.734582 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.734655 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.734709 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:55:37.736153 19395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:37.737743 19395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:37.743908 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.739748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:39367 (local address 127.0.0.1:39158)
0620 22:55:37.740028 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.740071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.740136 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.740302 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.740305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.740307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:37.740383 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.740385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.741728 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.741729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.742894 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.742896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.743626 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.743627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.743702 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.743715 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.743781 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.743832 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":11,"thread_start_us":57,"threads_started":1}
I20260620 22:55:37.743952 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.739858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.2.90.129:39367)
0620 22:55:37.740101 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.740105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.740116 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:55:37.740210 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.740212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.740226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.740297 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:37.740463 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.741634 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.742963 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.743558 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.743618 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.743678 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.743816 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.743873 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260620 22:55:37.744204 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.739845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.2.90.130:40283)
0620 22:55:37.740026 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.740031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.740216 (+ 185us) server_negotiation.cc:407] Connection header received
0620 22:55:37.740262 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.740264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.740279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.740350 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:37.740558 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.741331 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.742797 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.743713 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.743764 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.743808 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.743903 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.743939 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260620 22:55:37.743992 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.739751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:40283 (local address 127.0.0.1:43160)
0620 22:55:37.740029 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.740071 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.740137 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.740440 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.740441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.740442 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:37.740491 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.740493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.741456 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.741457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.742721 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.742724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.743776 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.743777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.743829 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.743832 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.743890 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.743917 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":56,"threads_started":1}
I20260620 22:55:37.744328 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.740114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.2.90.131:33059)
0620 22:55:37.740290 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.740294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.740396 (+ 102us) server_negotiation.cc:407] Connection header received
0620 22:55:37.740447 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.740450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.740463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.740528 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:37.740736 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.741760 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.742882 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.743555 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.743625 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.743678 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.744208 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.744250 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260620 22:55:37.744237 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.739948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:33059 (local address 127.0.0.1:48116)
0620 22:55:37.740333 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.740342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.740394 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.740598 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.740599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.740601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:37.740636 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.740638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.741856 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.741856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.742794 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.742796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.744092 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.744093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.744133 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.744136 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.744176 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.744199 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":30,"threads_started":1}
I20260620 22:55:37.744969 19520 tablet_service.cc:1467] Processing CreateTablet for tablet c52315f23470461ea6ddcb9595676aaa (DEFAULT_TABLE table=test-table [id=28718c7033be42a58eb5667e74850c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:37.744994 19654 tablet_service.cc:1467] Processing CreateTablet for tablet c52315f23470461ea6ddcb9595676aaa (DEFAULT_TABLE table=test-table [id=28718c7033be42a58eb5667e74850c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:37.745309 19520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52315f23470461ea6ddcb9595676aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:37.745362 19654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52315f23470461ea6ddcb9595676aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:37.745478 19788 tablet_service.cc:1467] Processing CreateTablet for tablet c52315f23470461ea6ddcb9595676aaa (DEFAULT_TABLE table=test-table [id=28718c7033be42a58eb5667e74850c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:37.745738 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52315f23470461ea6ddcb9595676aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:37.748472 19869 tablet_bootstrap.cc:492] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Bootstrap starting.
I20260620 22:55:37.748719 19868 tablet_bootstrap.cc:492] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: Bootstrap starting.
I20260620 22:55:37.749083 19870 tablet_bootstrap.cc:492] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Bootstrap starting.
I20260620 22:55:37.749172 19869 tablet_bootstrap.cc:654] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:37.749367 19868 tablet_bootstrap.cc:654] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:37.749713 19870 tablet_bootstrap.cc:654] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:37.749752 19869 log.cc:824] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:37.749837 19868 log.cc:824] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:37.750185 19870 log.cc:824] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:37.750490 19868 tablet_bootstrap.cc:492] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: No bootstrap required, opened a new log
I20260620 22:55:37.750520 19869 tablet_bootstrap.cc:492] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: No bootstrap required, opened a new log
I20260620 22:55:37.750561 19868 ts_tablet_manager.cc:1397] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:37.750572 19869 ts_tablet_manager.cc:1397] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:55:37.750866 19870 tablet_bootstrap.cc:492] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: No bootstrap required, opened a new log
I20260620 22:55:37.750929 19870 ts_tablet_manager.cc:1397] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:37.751940 19869 raft_consensus.cc:348] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.752080 19869 raft_consensus.cc:374] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:37.752112 19869 raft_consensus.cc:729] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbe119d9a8d43b2b7f6fb396177d30a, State: Initialized, Role: FOLLOWER
I20260620 22:55:37.752254 19869 consensus_queue.cc:260] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.752499 19869 ts_tablet_manager.cc:1428] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:55:37.752429 19868 raft_consensus.cc:348] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.752614 19720 heartbeater.cc:503] Master 127.2.90.190:41995 was elected leader, sending a full tablet report...
I20260620 22:55:37.752611 19868 raft_consensus.cc:374] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:37.752648 19868 raft_consensus.cc:729] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5897e621532b44368e7467f8890d23ee, State: Initialized, Role: FOLLOWER
I20260620 22:55:37.752777 19868 consensus_queue.cc:260] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.753053 19868 ts_tablet_manager.cc:1428] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:55:37.753176 19586 heartbeater.cc:503] Master 127.2.90.190:41995 was elected leader, sending a full tablet report...
I20260620 22:55:37.753152 19870 raft_consensus.cc:348] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.753306 19870 raft_consensus.cc:374] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:37.753357 19870 raft_consensus.cc:729] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd09c0708ac46268efd1877fa3bcd38, State: Initialized, Role: FOLLOWER
I20260620 22:55:37.753486 19870 consensus_queue.cc:260] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.753764 19854 heartbeater.cc:503] Master 127.2.90.190:41995 was elected leader, sending a full tablet report...
I20260620 22:55:37.753970 19870 ts_tablet_manager.cc:1428] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260620 22:55:37.816341 19721 tablet.cc:2369] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:37.890794 19875 raft_consensus.cc:482] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:37.890911 19875 raft_consensus.cc:504] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.891208 19875 leader_election.cc:288] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bbe119d9a8d43b2b7f6fb396177d30a (127.2.90.130:40283), ddd09c0708ac46268efd1877fa3bcd38 (127.2.90.131:33059)
I20260620 22:55:37.893970 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.891418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:46159 (local address 127.2.90.131:33059)
0620 22:55:37.891482 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.891484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.891495 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:55:37.891568 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.891570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.891579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.891634 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:37.891786 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.892293 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.893276 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.893653 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.893714 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.893723 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.893869 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.893908 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 22:55:37.894001 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.891381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:33059 (local address 127.2.90.129:46159)
0620 22:55:37.891447 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.891458 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.891497 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.891650 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.891652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.891654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:37.891722 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.891724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.892357 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.892358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.893193 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.893196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.893729 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.893730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.893803 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.893806 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.893848 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.893899 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 22:55:37.894011 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.891552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:47587 (local address 127.2.90.130:40283)
0620 22:55:37.891591 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.891593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.891618 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:55:37.891669 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.891670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.891681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.891747 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:37.891891 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.892384 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.893342 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.893705 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.893767 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.893775 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.893908 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.893944 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:55:37.894058 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.891381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:40283 (local address 127.2.90.129:47587)
0620 22:55:37.891577 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.891585 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:37.891623 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:37.891748 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:37.891748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:37.891750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:37.891818 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.891819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.892456 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.892456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.893246 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:37.893248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.893776 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.893777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:37.893849 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.893853 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:37.893896 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:55:37.893932 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":71,"thread_start_us":63,"threads_started":1}
I20260620 22:55:37.894361 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52315f23470461ea6ddcb9595676aaa" candidate_uuid: "5897e621532b44368e7467f8890d23ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd09c0708ac46268efd1877fa3bcd38" is_pre_election: true
I20260620 22:55:37.894359 19674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52315f23470461ea6ddcb9595676aaa" candidate_uuid: "5897e621532b44368e7467f8890d23ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" is_pre_election: true
I20260620 22:55:37.894510 19674 raft_consensus.cc:2454] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5897e621532b44368e7467f8890d23ee in term 0.
I20260620 22:55:37.894511 19808 raft_consensus.cc:2454] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5897e621532b44368e7467f8890d23ee in term 0.
I20260620 22:55:37.894714 19473 leader_election.cc:302] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbe119d9a8d43b2b7f6fb396177d30a, 5897e621532b44368e7467f8890d23ee; no voters:
I20260620 22:55:37.894891 19875 raft_consensus.cc:2781] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:37.895007 19875 raft_consensus.cc:482] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:37.895040 19875 raft_consensus.cc:3037] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:37.895773 19875 raft_consensus.cc:504] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.895895 19875 leader_election.cc:288] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [CANDIDATE]: Term 1 election: Requested vote from peers 0bbe119d9a8d43b2b7f6fb396177d30a (127.2.90.130:40283), ddd09c0708ac46268efd1877fa3bcd38 (127.2.90.131:33059)
I20260620 22:55:37.896035 19674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52315f23470461ea6ddcb9595676aaa" candidate_uuid: "5897e621532b44368e7467f8890d23ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a"
I20260620 22:55:37.896065 19808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52315f23470461ea6ddcb9595676aaa" candidate_uuid: "5897e621532b44368e7467f8890d23ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd09c0708ac46268efd1877fa3bcd38"
I20260620 22:55:37.896106 19674 raft_consensus.cc:3037] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:37.896136 19808 raft_consensus.cc:3037] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:37.896859 19674 raft_consensus.cc:2454] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5897e621532b44368e7467f8890d23ee in term 1.
I20260620 22:55:37.896888 19808 raft_consensus.cc:2454] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5897e621532b44368e7467f8890d23ee in term 1.
I20260620 22:55:37.897007 19473 leader_election.cc:302] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbe119d9a8d43b2b7f6fb396177d30a, 5897e621532b44368e7467f8890d23ee; no voters:
I20260620 22:55:37.897110 19875 raft_consensus.cc:2781] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:37.897265 19875 raft_consensus.cc:686] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 1 LEADER]: Becoming Leader. State: Replica: 5897e621532b44368e7467f8890d23ee, State: Running, Role: LEADER
I20260620 22:55:37.897351 19875 consensus_queue.cc:237] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } }
I20260620 22:55:37.898128 19395 catalog_manager.cc:5495] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 5897e621532b44368e7467f8890d23ee (127.2.90.129). New cstate: current_term: 1 leader_uuid: "5897e621532b44368e7467f8890d23ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5897e621532b44368e7467f8890d23ee" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 39367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 } health_report { overall_health: UNKNOWN } } }
W20260620 22:55:37.909932 19587 tablet.cc:2369] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:37.922892 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.920944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.2.90.129:39367)
0620 22:55:37.920988 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:37.920990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:37.920997 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:55:37.921028 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:37.921029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:37.921047 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:37.921103 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:37.921198 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.921643 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.922313 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:37.922482 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:37.922522 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:37.922617 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:37.922707 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:37.922757 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:37.922770 (+ 13us) server_negotiation.cc:299] Negotiation successful
0620 22:55:37.922784 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:55:37.924464 19674 raft_consensus.cc:1261] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 1 FOLLOWER]: Refusing update from remote peer 5897e621532b44368e7467f8890d23ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:37.924479 19808 raft_consensus.cc:1261] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 1 FOLLOWER]: Refusing update from remote peer 5897e621532b44368e7467f8890d23ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:37.924804 19878 consensus_queue.cc:1035] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbe119d9a8d43b2b7f6fb396177d30a" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:37.924938 19875 consensus_queue.cc:1035] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd09c0708ac46268efd1877fa3bcd38" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:37.927146 19881 mvcc.cc:205] Tried to move back new op lower bound from 7299056180936228864 to 7299056180828254208. Current Snapshot: MvccSnapshot[applied={T|T < 7299056180936228864}]
I20260620 22:55:37.929287 19395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:37.929369 19395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:37.930012 19395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eab658e6ae2846679f839519d30233c2: Sending DeleteTablet for 3 replicas of tablet c52315f23470461ea6ddcb9595676aaa
I20260620 22:55:37.930284 19520 tablet_service.cc:1514] Processing DeleteTablet for tablet c52315f23470461ea6ddcb9595676aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:37 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:39158
I20260620 22:55:37.930305 19654 tablet_service.cc:1514] Processing DeleteTablet for tablet c52315f23470461ea6ddcb9595676aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:37 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:43160
I20260620 22:55:37.930502 19890 tablet_replica.cc:330] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee: stopping tablet replica
I20260620 22:55:37.930485 19788 tablet_service.cc:1514] Processing DeleteTablet for tablet c52315f23470461ea6ddcb9595676aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:37 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:48116
I20260620 22:55:37.930593 19890 raft_consensus.cc:2229] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:55:37.930672 19890 raft_consensus.cc:2258] T c52315f23470461ea6ddcb9595676aaa P 5897e621532b44368e7467f8890d23ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:37.930738 19892 tablet_replica.cc:330] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: stopping tablet replica
I20260620 22:55:37.930800 19891 tablet_replica.cc:330] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: stopping tablet replica
I20260620 22:55:37.930814 19892 raft_consensus.cc:2229] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:37.930872 19891 raft_consensus.cc:2229] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:37.930889 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19457
I20260620 22:55:37.930910 19892 raft_consensus.cc:2258] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:37.930944 19891 raft_consensus.cc:2258] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:37.931671 19891 ts_tablet_manager.cc:1905] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:37.933104 19891 ts_tablet_manager.cc:1918] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:37.933166 19891 log.cc:1196] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/c52315f23470461ea6ddcb9595676aaa
I20260620 22:55:37.933434 19891 ts_tablet_manager.cc:1939] T c52315f23470461ea6ddcb9595676aaa P 0bbe119d9a8d43b2b7f6fb396177d30a: Deleting consensus metadata
I20260620 22:55:37.933763 19379 catalog_manager.cc:4841] TS 0bbe119d9a8d43b2b7f6fb396177d30a (127.2.90.130:40283): tablet c52315f23470461ea6ddcb9595676aaa (table test-table [id=28718c7033be42a58eb5667e74850c10]) successfully deleted
W20260620 22:55:37.935722 19382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:39367 (error 108)
I20260620 22:55:37.935901 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19591
I20260620 22:55:37.936095 19892 ts_tablet_manager.cc:1905] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:37.936497 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:37.936059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:39367 (local address 127.0.0.1:39176)
0620 22:55:37.936422 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:37.936465 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:39367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
W20260620 22:55:37.936615 19382 catalog_manager.cc:4584] TS 5897e621532b44368e7467f8890d23ee (127.2.90.129:39367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c52315f23470461ea6ddcb9595676aaa: Network error: Client connection negotiation failed: client connection to 127.2.90.129:39367: connect: Connection refused (error 111)
I20260620 22:55:37.937520 19892 ts_tablet_manager.cc:1918] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:37.937580 19892 log.cc:1196] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/c52315f23470461ea6ddcb9595676aaa
I20260620 22:55:37.937820 19892 ts_tablet_manager.cc:1939] T c52315f23470461ea6ddcb9595676aaa P ddd09c0708ac46268efd1877fa3bcd38: Deleting consensus metadata
I20260620 22:55:37.938222 19382 catalog_manager.cc:4841] TS ddd09c0708ac46268efd1877fa3bcd38 (127.2.90.131:33059): tablet c52315f23470461ea6ddcb9595676aaa (table test-table [id=28718c7033be42a58eb5667e74850c10]) successfully deleted
I20260620 22:55:37.941560 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19725
I20260620 22:55:37.946004 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19363
2026-06-20T22:55:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1880 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T22:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:37Z Disabled control of system clock
I20260620 22:55:37.971632 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:33221
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:38.048835 19897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:38.049042 19897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:38.049069 19897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:38.050189 19897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:38.050227 19897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:38.050244 19897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:38.050259 19897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:38.050274 19897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:38.051573 19897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:33221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:38.051812 19897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:38.052070 19897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:38.056515 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:38.056584 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.056550 19897 server_base.cc:1034] running on GCE node
W20260620 22:55:38.056723 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.056951 19897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:38.057226 19897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:38.058372 19897 hybrid_clock.cc:648] HybridClock initialized: now 1781996138058355 us; error 31 us; skew 500 ppm
I20260620 22:55:38.059480 19897 webserver.cc:460] Webserver started at http://127.2.90.190:37605/ using document root <none> and password file <none>
I20260620 22:55:38.059701 19897 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:38.059749 19897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:38.059880 19897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:38.060932 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "2d2f169818214ab38ef6e17809b72daf"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "2dd3fb43fe056e9e8d66991131c039aa"
server_key_iv: "3e947e95c834fe310f8bb83082a7879a"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.061237 19897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "2d2f169818214ab38ef6e17809b72daf"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "2dd3fb43fe056e9e8d66991131c039aa"
server_key_iv: "3e947e95c834fe310f8bb83082a7879a"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.062366 19897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:38.063043 19911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.063203 19897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:38.063282 19897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "2d2f169818214ab38ef6e17809b72daf"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "2dd3fb43fe056e9e8d66991131c039aa"
server_key_iv: "3e947e95c834fe310f8bb83082a7879a"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.063344 19897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:38.079921 19897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:38.080209 19897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:38.080333 19897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:38.083915 19897 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:33221
I20260620 22:55:38.083951 19963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:33221 every 8 connection(s)
I20260620 22:55:38.084254 19897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:38.085197 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:38.085491 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19897
I20260620 22:55:38.085597 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:38.085772 2410 external_mini_cluster.cc:1428] Setting key 07f9d169d42f44b4a74cb33b1bea1380
I20260620 22:55:38.087739 19964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf: Bootstrap starting.
I20260620 22:55:38.088305 19964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:38.088629 19964 log.cc:824] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:38.089293 19964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf: No bootstrap required, opened a new log
I20260620 22:55:38.090835 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.086262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.2.90.190:33221)
0620 22:55:38.086443 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.086450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.086474 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:55:38.086883 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.086889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.086950 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.087058 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:38.087269 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.088129 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.089136 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.089555 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.089640 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.090262 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:38.090265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:38.090266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:38.090414 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:38.090445 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:38.090455 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:38.090501 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.090581 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.090642 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 22:55:38.091277 19964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2f169818214ab38ef6e17809b72daf" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33221 } }
I20260620 22:55:38.091413 19964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:38.091444 19964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2f169818214ab38ef6e17809b72daf, State: Initialized, Role: FOLLOWER
I20260620 22:55:38.091549 19964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2f169818214ab38ef6e17809b72daf" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33221 } }
I20260620 22:55:38.091609 19964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:38.091637 19964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:38.091681 19964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:38.092154 19964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2f169818214ab38ef6e17809b72daf" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33221 } }
I20260620 22:55:38.092255 19964 leader_election.cc:302] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d2f169818214ab38ef6e17809b72daf; no voters:
I20260620 22:55:38.092413 19964 leader_election.cc:288] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:38.092447 19969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:38.092576 19969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [term 1 LEADER]: Becoming Leader. State: Replica: 2d2f169818214ab38ef6e17809b72daf, State: Running, Role: LEADER
I20260620 22:55:38.092654 19964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:38.092703 19969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2f169818214ab38ef6e17809b72daf" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33221 } }
I20260620 22:55:38.093022 19971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2f169818214ab38ef6e17809b72daf. Latest consensus state: current_term: 1 leader_uuid: "2d2f169818214ab38ef6e17809b72daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2f169818214ab38ef6e17809b72daf" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33221 } } }
I20260620 22:55:38.093144 19971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:38.093021 19970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d2f169818214ab38ef6e17809b72daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2f169818214ab38ef6e17809b72daf" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 33221 } } }
I20260620 22:55:38.093353 19970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:38.093602 19976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:38.094015 19976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:38.095405 19976 catalog_manager.cc:1347] Generated new cluster ID: 3910b81b2c934ae39a7970f93b62176e
I20260620 22:55:38.095448 19976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:38.123709 19976 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:38.124235 19976 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:38.126255 19976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf: Generated new TSK 0
I20260620 22:55:38.126418 19976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:38.132015 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:33221
--builtin_ntp_servers=127.2.90.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:38.209625 19988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:38.209818 19988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:38.209856 19988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:38.210952 19988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:38.210989 19988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:38.211021 19988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:38.212322 19988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:33221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:38.212549 19988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:38.212831 19988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:38.217307 19994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:38.217344 19996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.217504 19988 server_base.cc:1034] running on GCE node
W20260620 22:55:38.217506 19993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.217726 19988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:38.217943 19988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:38.219074 19988 hybrid_clock.cc:648] HybridClock initialized: now 1781996138219058 us; error 27 us; skew 500 ppm
I20260620 22:55:38.220171 19988 webserver.cc:460] Webserver started at http://127.2.90.129:43947/ using document root <none> and password file <none>
I20260620 22:55:38.220395 19988 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:38.220443 19988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:38.220568 19988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:38.221652 19988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "e8ca462d35a14aedad42a5a96128bed8"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "912c63c23c66eaf879a7b41a525f6379"
server_key_iv: "850d82075604505931536da27c087737"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.221982 19988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "e8ca462d35a14aedad42a5a96128bed8"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "912c63c23c66eaf879a7b41a525f6379"
server_key_iv: "850d82075604505931536da27c087737"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.223112 19988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:38.223716 20002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.223878 19988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:38.223955 19988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "e8ca462d35a14aedad42a5a96128bed8"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "912c63c23c66eaf879a7b41a525f6379"
server_key_iv: "850d82075604505931536da27c087737"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.224021 19988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:38.238749 19988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:38.239027 19988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:38.239149 19988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:38.239354 19988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:38.239677 19988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:38.239709 19988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.239738 19988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:38.239755 19988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.244920 19988 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:40335
I20260620 22:55:38.244992 20115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:40335 every 8 connection(s)
I20260620 22:55:38.245262 19988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:55:38.246408 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 19988
I20260620 22:55:38.246503 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:55:38.246683 2410 external_mini_cluster.cc:1428] Setting key bb0649e8164cc0d2538d9e3078754953
I20260620 22:55:38.247550 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:33221
--builtin_ntp_servers=127.2.90.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:55:38.250602 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.245908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:55307 (local address 127.2.90.190:33221)
0620 22:55:38.245967 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.245969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.246176 (+ 207us) server_negotiation.cc:407] Connection header received
0620 22:55:38.246704 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.246705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.246721 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.246800 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:38.247623 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.248292 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.249269 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.249552 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.249616 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.250351 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:38.250353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:38.250353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:38.250357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:38.250370 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:38.250373 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:38.250416 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.250495 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.250525 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 22:55:38.250784 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.245885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:33221 (local address 127.2.90.129:55307)
0620 22:55:38.246059 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.246100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.246603 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.246886 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.246891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.246902 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:38.247536 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.247542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.248372 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.248373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.249195 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.249197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.249629 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.249629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.249732 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.250214 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:38.250214 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:38.250242 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:38.250264 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:38.250422 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:38.250427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:38.250430 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.250491 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.250587 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260620 22:55:38.251279 20116 heartbeater.cc:347] Connected to a master server at 127.2.90.190:33221
I20260620 22:55:38.251389 20116 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:38.251579 20116 heartbeater.cc:511] Master 127.2.90.190:33221 requested a full tablet report, sending...
I20260620 22:55:38.252040 19928 ts_manager.cc:194] Registered new tserver with Master: e8ca462d35a14aedad42a5a96128bed8 (127.2.90.129:40335)
I20260620 22:55:38.252638 19928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.129:55307
W20260620 22:55:38.325637 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:38.325830 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:38.325850 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:38.327034 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:38.327073 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:38.327097 20119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:38.328424 20119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:33221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:38.328630 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:38.328913 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:38.333314 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:38.333338 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.333555 20119 server_base.cc:1034] running on GCE node
W20260620 22:55:38.333598 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.333786 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:38.334003 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:38.335140 20119 hybrid_clock.cc:648] HybridClock initialized: now 1781996138335127 us; error 29 us; skew 500 ppm
I20260620 22:55:38.336208 20119 webserver.cc:460] Webserver started at http://127.2.90.130:40925/ using document root <none> and password file <none>
I20260620 22:55:38.336412 20119 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:38.336460 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:38.336575 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:38.337605 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "c2a2e1909cc8475cbfa346dc576256b1"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "cd6b9dfc09ef63a5cdb1900f2846fc3d"
server_key_iv: "e85db76eed4cddafc4303dc155153dc2"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.337942 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "c2a2e1909cc8475cbfa346dc576256b1"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "cd6b9dfc09ef63a5cdb1900f2846fc3d"
server_key_iv: "e85db76eed4cddafc4303dc155153dc2"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.339063 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:38.339648 20133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.339789 20119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:38.339870 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "c2a2e1909cc8475cbfa346dc576256b1"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "cd6b9dfc09ef63a5cdb1900f2846fc3d"
server_key_iv: "e85db76eed4cddafc4303dc155153dc2"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.339933 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:38.360211 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:38.360502 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:38.360633 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:38.360858 20119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:38.361166 20119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:38.361199 20119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.361227 20119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:38.361248 20119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.366514 20119 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:35347
I20260620 22:55:38.366552 20246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:35347 every 8 connection(s)
I20260620 22:55:38.366853 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:38.371627 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.367514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:37411 (local address 127.2.90.190:33221)
0620 22:55:38.367570 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.367572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.367772 (+ 200us) server_negotiation.cc:407] Connection header received
0620 22:55:38.368254 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.368255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.368266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.368313 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:38.368990 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.369485 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.370413 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.370599 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.370653 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.371378 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:38.371380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:38.371380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:38.371385 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:38.371400 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:38.371404 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:38.371444 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.371519 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.371551 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 22:55:38.371801 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.367515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:33221 (local address 127.2.90.130:37411)
0620 22:55:38.367662 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.367695 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.368163 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.368380 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.368383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.368391 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:38.368903 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.368909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.369571 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.369572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.370332 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.370334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.370670 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.370671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.370787 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.371249 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:38.371249 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:38.371278 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:38.371300 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:38.371450 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:38.371455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:38.371457 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.371516 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.371613 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260620 22:55:38.372213 20247 heartbeater.cc:347] Connected to a master server at 127.2.90.190:33221
I20260620 22:55:38.372269 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20119
I20260620 22:55:38.372303 20247 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:38.372375 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:55:38.372512 20247 heartbeater.cc:511] Master 127.2.90.190:33221 requested a full tablet report, sending...
I20260620 22:55:38.372560 2410 external_mini_cluster.cc:1428] Setting key e741b7d623c5498fe79bba25026cd617
I20260620 22:55:38.372871 19928 ts_manager.cc:194] Registered new tserver with Master: c2a2e1909cc8475cbfa346dc576256b1 (127.2.90.130:35347)
I20260620 22:55:38.373333 19928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.130:37411
I20260620 22:55:38.373588 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:33221
--builtin_ntp_servers=127.2.90.148:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:38.450819 20250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:38.451014 20250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:38.451035 20250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:38.452103 20250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:38.452134 20250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:38.452157 20250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:38.453488 20250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:33221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:38.453661 20250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:38.453888 20250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:38.458272 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:38.458304 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:38.458376 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:38.458423 20250 server_base.cc:1034] running on GCE node
I20260620 22:55:38.458690 20250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:38.458909 20250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:38.460039 20250 hybrid_clock.cc:648] HybridClock initialized: now 1781996138460027 us; error 29 us; skew 500 ppm
I20260620 22:55:38.461174 20250 webserver.cc:460] Webserver started at http://127.2.90.131:39777/ using document root <none> and password file <none>
I20260620 22:55:38.461376 20250 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:38.461418 20250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:38.461539 20250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:38.462546 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "225261f08b6f4b3f9443a2cacf2886b4"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "9171ab6e147609ff5f5f5f32860e5bb0"
server_key_iv: "f6f8b0acdef806af19de3ec2f2920923"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.462868 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "225261f08b6f4b3f9443a2cacf2886b4"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "9171ab6e147609ff5f5f5f32860e5bb0"
server_key_iv: "f6f8b0acdef806af19de3ec2f2920923"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.463976 20250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:55:38.464577 20264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.464727 20250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:38.464804 20250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "225261f08b6f4b3f9443a2cacf2886b4"
format_stamp: "Formatted at 2026-06-20 22:55:38 on dist-test-slave-4mtw"
server_key: "9171ab6e147609ff5f5f5f32860e5bb0"
server_key_iv: "f6f8b0acdef806af19de3ec2f2920923"
server_key_version: "encryptionkey@0"
I20260620 22:55:38.464876 20250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:38.475385 20250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:38.475631 20250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:38.475747 20250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:38.475939 20250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:38.476223 20250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:38.476254 20250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.476282 20250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:38.476305 20250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:38.481541 20250 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:35669
I20260620 22:55:38.481598 20377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:35669 every 8 connection(s)
I20260620 22:55:38.481880 20250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:38.486263 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.482461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:34983 (local address 127.2.90.190:33221)
0620 22:55:38.482515 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.482517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.482706 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:55:38.483162 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.483163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.483173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.483220 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:38.483798 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.484270 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.485211 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.485403 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.485453 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.486065 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:38.486067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:38.486067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:38.486071 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:38.486085 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:38.486088 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:38.486123 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.486182 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.486205 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 22:55:38.486440 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.482457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:33221 (local address 127.2.90.131:34983)
0620 22:55:38.482603 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.482634 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.483088 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.483271 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.483274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.483282 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:38.483740 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.483744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.484327 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.484327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.485138 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.485141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.485459 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.485460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.485557 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.485956 (+ 399us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:38.485956 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:38.485982 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:38.486001 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:38.486130 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:38.486132 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:38.486134 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.486178 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.486251 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260620 22:55:38.486819 20378 heartbeater.cc:347] Connected to a master server at 127.2.90.190:33221
I20260620 22:55:38.486907 20378 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:38.487072 20378 heartbeater.cc:511] Master 127.2.90.190:33221 requested a full tablet report, sending...
I20260620 22:55:38.487389 19928 ts_manager.cc:194] Registered new tserver with Master: 225261f08b6f4b3f9443a2cacf2886b4 (127.2.90.131:35669)
I20260620 22:55:38.487730 19928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.131:34983
I20260620 22:55:38.487748 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20250
I20260620 22:55:38.487833 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:55:38.488011 2410 external_mini_cluster.cc:1428] Setting key bb5b81443e5c23d575757518ac24719a
I20260620 22:55:38.488819 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:38.492518 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.490034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.2.90.190:33221)
0620 22:55:38.490089 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.490091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.490178 (+ 87us) server_negotiation.cc:407] Connection header received
0620 22:55:38.490237 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.490238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.490246 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.490290 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:38.490425 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.490872 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.491548 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.491736 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.491785 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.492288 (+ 503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:38.492289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:38.492289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:38.492293 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:38.492306 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:38.492309 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:38.492379 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.492430 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.492450 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:55:38.493780 19928 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:38.495213 19928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:38.501202 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.497156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.2.90.129:40335)
0620 22:55:38.497483 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.497487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.497496 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:38.497533 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.497536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.497550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.497630 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:38.497784 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.498890 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.500334 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.500849 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.500920 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.501009 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.501090 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.501125 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":76,"threads_started":1}
I20260620 22:55:38.501194 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.497075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:40335 (local address 127.0.0.1:40152)
0620 22:55:38.497219 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.497256 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.497313 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.497639 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.497641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.497646 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:38.497712 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.497714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.498957 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.498957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.500243 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.500244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.500930 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.500931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.501005 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.501018 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.501077 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.501125 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":77,"thread_start_us":43,"threads_started":1}
I20260620 22:55:38.501523 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.497075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35347 (local address 127.0.0.1:41418)
0620 22:55:38.497323 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.497333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.497383 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.497895 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.497896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.497898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:38.497932 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.497934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.499130 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.499131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.500216 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.500218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.501320 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.501321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.501378 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.501382 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.501430 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.501463 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260620 22:55:38.501775 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.497298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.2.90.130:35347)
0620 22:55:38.497751 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.497755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.497763 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:38.497796 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.497798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.497813 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.497890 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:38.497986 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.499063 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.500295 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.501028 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.501090 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.501167 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.501658 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.501700 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":70,"threads_started":1}
I20260620 22:55:38.501839 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.497082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:35669 (local address 127.0.0.1:55908)
0620 22:55:38.497450 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.497458 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.497503 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.497638 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.497641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.497643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:38.497712 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.497715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.499741 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.499742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.500701 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.500704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.501613 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.501615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.501675 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.501679 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.501729 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.501765 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260620 22:55:38.501885 20050 tablet_service.cc:1467] Processing CreateTablet for tablet e67754aad84e40458754e8c1d5db2ffa (DEFAULT_TABLE table=test-table [id=6f99d3549d164176ba33968b66c08616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:38.502128 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.497259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.2.90.131:35669)
0620 22:55:38.497401 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.497406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.497502 (+ 96us) server_negotiation.cc:407] Connection header received
0620 22:55:38.497541 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.497543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.497555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.497629 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:38.498984 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.499662 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.500788 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.501451 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.501509 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.501577 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.501742 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.501778 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260620 22:55:38.502228 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67754aad84e40458754e8c1d5db2ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:38.502415 20181 tablet_service.cc:1467] Processing CreateTablet for tablet e67754aad84e40458754e8c1d5db2ffa (DEFAULT_TABLE table=test-table [id=6f99d3549d164176ba33968b66c08616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:38.502657 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67754aad84e40458754e8c1d5db2ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:38.502686 20312 tablet_service.cc:1467] Processing CreateTablet for tablet e67754aad84e40458754e8c1d5db2ffa (DEFAULT_TABLE table=test-table [id=6f99d3549d164176ba33968b66c08616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:38.502985 20312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67754aad84e40458754e8c1d5db2ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:38.504868 20392 tablet_bootstrap.cc:492] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Bootstrap starting.
I20260620 22:55:38.505393 20393 tablet_bootstrap.cc:492] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: Bootstrap starting.
I20260620 22:55:38.505436 20392 tablet_bootstrap.cc:654] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:38.505725 20392 log.cc:824] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:38.505940 20394 tablet_bootstrap.cc:492] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Bootstrap starting.
I20260620 22:55:38.505992 20393 tablet_bootstrap.cc:654] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:38.506315 20393 log.cc:824] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:38.506583 20394 tablet_bootstrap.cc:654] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:38.506646 20392 tablet_bootstrap.cc:492] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: No bootstrap required, opened a new log
I20260620 22:55:38.506709 20392 ts_tablet_manager.cc:1397] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:55:38.506877 20394 log.cc:824] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:38.506915 20393 tablet_bootstrap.cc:492] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: No bootstrap required, opened a new log
I20260620 22:55:38.506956 20393 ts_tablet_manager.cc:1397] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:38.507587 20394 tablet_bootstrap.cc:492] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: No bootstrap required, opened a new log
I20260620 22:55:38.507637 20394 ts_tablet_manager.cc:1397] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:38.508641 20393 raft_consensus.cc:348] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.508828 20393 raft_consensus.cc:374] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:38.508893 20393 raft_consensus.cc:729] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ca462d35a14aedad42a5a96128bed8, State: Initialized, Role: FOLLOWER
I20260620 22:55:38.509020 20393 consensus_queue.cc:260] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.509109 20392 raft_consensus.cc:348] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.509254 20393 ts_tablet_manager.cc:1428] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:55:38.509262 20392 raft_consensus.cc:374] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:38.509289 20116 heartbeater.cc:503] Master 127.2.90.190:33221 was elected leader, sending a full tablet report...
I20260620 22:55:38.509321 20392 raft_consensus.cc:729] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a2e1909cc8475cbfa346dc576256b1, State: Initialized, Role: FOLLOWER
I20260620 22:55:38.509425 20392 consensus_queue.cc:260] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.509660 20392 ts_tablet_manager.cc:1428] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:38.509897 20247 heartbeater.cc:503] Master 127.2.90.190:33221 was elected leader, sending a full tablet report...
I20260620 22:55:38.510262 20394 raft_consensus.cc:348] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.510422 20394 raft_consensus.cc:374] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:38.510461 20394 raft_consensus.cc:729] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225261f08b6f4b3f9443a2cacf2886b4, State: Initialized, Role: FOLLOWER
I20260620 22:55:38.510566 20394 consensus_queue.cc:260] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.510766 20394 ts_tablet_manager.cc:1428] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:38.510802 20378 heartbeater.cc:503] Master 127.2.90.190:33221 was elected leader, sending a full tablet report...
I20260620 22:55:38.614926 20400 raft_consensus.cc:482] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:38.615044 20400 raft_consensus.cc:504] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.615406 20400 leader_election.cc:288] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8ca462d35a14aedad42a5a96128bed8 (127.2.90.129:40335), c2a2e1909cc8475cbfa346dc576256b1 (127.2.90.130:35347)
W20260620 22:55:38.617329 20248 tablet.cc:2369] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:38.618978 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.615629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:50341 (local address 127.2.90.130:35347)
0620 22:55:38.615670 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.615673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.615812 (+ 139us) server_negotiation.cc:407] Connection header received
0620 22:55:38.615857 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.615858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.615866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.615901 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:38.616030 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.616744 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.618108 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.618652 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.618718 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.618724 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.618874 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.618912 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:55:38.618978 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.615646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:53047 (local address 127.2.90.129:40335)
0620 22:55:38.615695 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.615697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.615706 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:38.615732 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.615733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.615744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.615796 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:38.615948 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.616704 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.618109 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.618652 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.618716 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.618724 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.618885 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.618919 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 22:55:38.618990 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.615555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:40335 (local address 127.2.90.131:53047)
0620 22:55:38.615606 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.615616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.615653 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.615807 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.615808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.615810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:38.615883 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.615885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.616766 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.616766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.618016 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.618019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.618725 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.618725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.618808 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.618813 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.618869 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.618903 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":38}
I20260620 22:55:38.619001 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.615587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35347 (local address 127.2.90.131:50341)
0620 22:55:38.615743 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.615750 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:38.615800 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:38.615918 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:38.615919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:38.615920 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:38.615973 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.615974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.616803 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.616803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.618017 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:38.618019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.618724 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.618724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:38.618808 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.618813 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:55:38.618864 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:55:38.618902 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":32,"thread_start_us":88,"threads_started":1}
I20260620 22:55:38.619333 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67754aad84e40458754e8c1d5db2ffa" candidate_uuid: "225261f08b6f4b3f9443a2cacf2886b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ca462d35a14aedad42a5a96128bed8" is_pre_election: true
I20260620 22:55:38.619388 20201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67754aad84e40458754e8c1d5db2ffa" candidate_uuid: "225261f08b6f4b3f9443a2cacf2886b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a2e1909cc8475cbfa346dc576256b1" is_pre_election: true
I20260620 22:55:38.619472 20070 raft_consensus.cc:2454] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225261f08b6f4b3f9443a2cacf2886b4 in term 0.
I20260620 22:55:38.619515 20201 raft_consensus.cc:2454] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225261f08b6f4b3f9443a2cacf2886b4 in term 0.
I20260620 22:55:38.619726 20265 leader_election.cc:302] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225261f08b6f4b3f9443a2cacf2886b4, c2a2e1909cc8475cbfa346dc576256b1; no voters:
I20260620 22:55:38.619843 20400 raft_consensus.cc:2781] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:38.619895 20400 raft_consensus.cc:482] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:38.619916 20400 raft_consensus.cc:3037] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:38.620574 20400 raft_consensus.cc:504] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.620728 20400 leader_election.cc:288] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [CANDIDATE]: Term 1 election: Requested vote from peers e8ca462d35a14aedad42a5a96128bed8 (127.2.90.129:40335), c2a2e1909cc8475cbfa346dc576256b1 (127.2.90.130:35347)
I20260620 22:55:38.620857 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67754aad84e40458754e8c1d5db2ffa" candidate_uuid: "225261f08b6f4b3f9443a2cacf2886b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ca462d35a14aedad42a5a96128bed8"
I20260620 22:55:38.620870 20201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67754aad84e40458754e8c1d5db2ffa" candidate_uuid: "225261f08b6f4b3f9443a2cacf2886b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a2e1909cc8475cbfa346dc576256b1"
I20260620 22:55:38.620926 20070 raft_consensus.cc:3037] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:38.620927 20201 raft_consensus.cc:3037] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:38.621580 20201 raft_consensus.cc:2454] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225261f08b6f4b3f9443a2cacf2886b4 in term 1.
I20260620 22:55:38.621589 20070 raft_consensus.cc:2454] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225261f08b6f4b3f9443a2cacf2886b4 in term 1.
I20260620 22:55:38.621729 20265 leader_election.cc:302] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225261f08b6f4b3f9443a2cacf2886b4, c2a2e1909cc8475cbfa346dc576256b1; no voters:
I20260620 22:55:38.621811 20400 raft_consensus.cc:2781] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:38.621949 20400 raft_consensus.cc:686] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 1 LEADER]: Becoming Leader. State: Replica: 225261f08b6f4b3f9443a2cacf2886b4, State: Running, Role: LEADER
I20260620 22:55:38.622027 20400 consensus_queue.cc:237] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } }
I20260620 22:55:38.622783 19927 catalog_manager.cc:5495] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 225261f08b6f4b3f9443a2cacf2886b4 (127.2.90.131). New cstate: current_term: 1 leader_uuid: "225261f08b6f4b3f9443a2cacf2886b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "225261f08b6f4b3f9443a2cacf2886b4" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 35669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 } health_report { overall_health: UNKNOWN } } }
I20260620 22:55:38.645052 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.642817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.2.90.131:35669)
0620 22:55:38.642885 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:38.642887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:38.642894 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:55:38.642949 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:38.642950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:38.642966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:38.643021 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:38.643171 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.643620 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.644350 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:38.644527 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:38.644571 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:38.644766 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:38.644871 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:38.644931 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:38.644947 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 22:55:38.644961 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 22:55:38.646696 20070 raft_consensus.cc:1261] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8 [term 1 FOLLOWER]: Refusing update from remote peer 225261f08b6f4b3f9443a2cacf2886b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:38.646698 20201 raft_consensus.cc:1261] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 1 FOLLOWER]: Refusing update from remote peer 225261f08b6f4b3f9443a2cacf2886b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:38.647050 20402 consensus_queue.cc:1035] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a2e1909cc8475cbfa346dc576256b1" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:38.647169 20400 consensus_queue.cc:1035] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ca462d35a14aedad42a5a96128bed8" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:38.649438 20405 mvcc.cc:205] Tried to move back new op lower bound from 7299056183894548480 to 7299056183796420608. Current Snapshot: MvccSnapshot[applied={T|T < 7299056183894548480}]
I20260620 22:55:38.649587 20406 mvcc.cc:205] Tried to move back new op lower bound from 7299056183894548480 to 7299056183796420608. Current Snapshot: MvccSnapshot[applied={T|T < 7299056183894548480}]
I20260620 22:55:38.649654 20407 mvcc.cc:205] Tried to move back new op lower bound from 7299056183894548480 to 7299056183796420608. Current Snapshot: MvccSnapshot[applied={T|T < 7299056183894548480}]
I20260620 22:55:38.651482 19928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:38.651578 19928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:38.652251 19928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d2f169818214ab38ef6e17809b72daf: Sending DeleteTablet for 3 replicas of tablet e67754aad84e40458754e8c1d5db2ffa
I20260620 22:55:38.652555 20312 tablet_service.cc:1514] Processing DeleteTablet for tablet e67754aad84e40458754e8c1d5db2ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:38 UTC) from {username='slave'} at 127.0.0.1:55908
I20260620 22:55:38.652661 20181 tablet_service.cc:1514] Processing DeleteTablet for tablet e67754aad84e40458754e8c1d5db2ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:38 UTC) from {username='slave'} at 127.0.0.1:41418
I20260620 22:55:38.652771 20050 tablet_service.cc:1514] Processing DeleteTablet for tablet e67754aad84e40458754e8c1d5db2ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:38 UTC) from {username='slave'} at 127.0.0.1:40152
I20260620 22:55:38.652968 20418 tablet_replica.cc:330] T e67754aad84e40458754e8c1d5db2ffa P e8ca462d35a14aedad42a5a96128bed8: stopping tablet replica
I20260620 22:55:38.652971 20416 tablet_replica.cc:330] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: stopping tablet replica
I20260620 22:55:38.653028 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19988
I20260620 22:55:38.653061 20416 raft_consensus.cc:2229] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:55:38.653143 20416 raft_consensus.cc:2258] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:38.653829 20416 ts_tablet_manager.cc:1905] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:38.654800 20417 tablet_replica.cc:330] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: stopping tablet replica
I20260620 22:55:38.654945 20417 raft_consensus.cc:2229] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:38.655261 20417 raft_consensus.cc:2258] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:38.655463 20416 ts_tablet_manager.cc:1918] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:38.655532 20416 log.cc:1196] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/e67754aad84e40458754e8c1d5db2ffa
I20260620 22:55:38.655746 20417 ts_tablet_manager.cc:1905] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:38.655831 20416 ts_tablet_manager.cc:1939] T e67754aad84e40458754e8c1d5db2ffa P 225261f08b6f4b3f9443a2cacf2886b4: Deleting consensus metadata
I20260620 22:55:38.656231 19914 catalog_manager.cc:4841] TS 225261f08b6f4b3f9443a2cacf2886b4 (127.2.90.131:35669): tablet e67754aad84e40458754e8c1d5db2ffa (table test-table [id=6f99d3549d164176ba33968b66c08616]) successfully deleted
I20260620 22:55:38.657135 20417 ts_tablet_manager.cc:1918] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:38.657193 20417 log.cc:1196] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/e67754aad84e40458754e8c1d5db2ffa
I20260620 22:55:38.657426 20417 ts_tablet_manager.cc:1939] T e67754aad84e40458754e8c1d5db2ffa P c2a2e1909cc8475cbfa346dc576256b1: Deleting consensus metadata
I20260620 22:55:38.657752 19912 catalog_manager.cc:4841] TS c2a2e1909cc8475cbfa346dc576256b1 (127.2.90.130:35347): tablet e67754aad84e40458754e8c1d5db2ffa (table test-table [id=6f99d3549d164176ba33968b66c08616]) successfully deleted
W20260620 22:55:38.659050 19915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:40335 (error 108)
I20260620 22:55:38.659228 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20119
I20260620 22:55:38.660128 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:38.659983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:40335 (local address 127.0.0.1:40154)
0620 22:55:38.660045 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:38.660094 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:40335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 22:55:38.660333 19915 catalog_manager.cc:4584] TS e8ca462d35a14aedad42a5a96128bed8 (127.2.90.129:40335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e67754aad84e40458754e8c1d5db2ffa: Network error: Client connection negotiation failed: client connection to 127.2.90.129:40335: connect: Connection refused (error 111)
I20260620 22:55:38.663592 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20250
I20260620 22:55:38.667989 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 19897
2026-06-20T22:55:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:38 dist-test-slave-4mtw krb5kdc[20422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:38 dist-test-slave-4mtw krb5kdc[20422](info): set up 2 sockets
Jun 20 22:55:38 dist-test-slave-4mtw krb5kdc[20422](info): commencing operation
krb5kdc: starting...
W20260620 22:55:39.608129 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:39 dist-test-slave-4mtw krb5kdc[20422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:55:39.712278 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42835
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:42835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:39.789469 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:39.789655 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:39.789675 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:39.790763 20435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:39.790802 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:39.790814 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:39.790845 20435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:39.790865 20435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:39.792303 20435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:42835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:42835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:39.792517 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:39.792733 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:39.797034 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:39.797055 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:39.797057 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:39.797380 20435 server_base.cc:1034] running on GCE node
I20260620 22:55:39.797570 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:39.797782 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:39.798916 20435 hybrid_clock.cc:648] HybridClock initialized: now 1781996139798899 us; error 34 us; skew 500 ppm
Jun 20 22:55:39 dist-test-slave-4mtw krb5kdc[20422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996139, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:39.801074 20435 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:55:39.801431 20435 webserver.cc:460] Webserver started at http://127.2.90.190:33915/ using document root <none> and password file <none>
I20260620 22:55:39.801604 20435 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:39.801640 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:39.801724 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:39.802644 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "ccd908dc2b2847568e36b7b8f39d798a"
format_stamp: "Formatted at 2026-06-20 22:55:39 on dist-test-slave-4mtw"
server_key: "9c79f44c1f4f78c48dcfff7b0a313efd"
server_key_iv: "2d05a9174685d6ecbb1265aa82679ca0"
server_key_version: "encryptionkey@0"
I20260620 22:55:39.802912 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "ccd908dc2b2847568e36b7b8f39d798a"
format_stamp: "Formatted at 2026-06-20 22:55:39 on dist-test-slave-4mtw"
server_key: "9c79f44c1f4f78c48dcfff7b0a313efd"
server_key_iv: "2d05a9174685d6ecbb1265aa82679ca0"
server_key_version: "encryptionkey@0"
I20260620 22:55:39.803979 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:39.804528 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:39.804670 20435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:39.804777 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "ccd908dc2b2847568e36b7b8f39d798a"
format_stamp: "Formatted at 2026-06-20 22:55:39 on dist-test-slave-4mtw"
server_key: "9c79f44c1f4f78c48dcfff7b0a313efd"
server_key_iv: "2d05a9174685d6ecbb1265aa82679ca0"
server_key_version: "encryptionkey@0"
I20260620 22:55:39.804831 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:39.823195 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:39.825235 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:39.825337 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:39.828733 20435 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:42835
I20260620 22:55:39.828795 20502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:42835 every 8 connection(s)
I20260620 22:55:39.829075 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:39.829829 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:39.831911 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a: Bootstrap starting.
I20260620 22:55:39.832433 20503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:39.832695 20503 log.cc:824] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:39.833304 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a: No bootstrap required, opened a new log
I20260620 22:55:39.834793 20503 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd908dc2b2847568e36b7b8f39d798a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42835 } }
I20260620 22:55:39.834928 20503 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:39.834960 20503 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd908dc2b2847568e36b7b8f39d798a, State: Initialized, Role: FOLLOWER
I20260620 22:55:39.835083 20503 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd908dc2b2847568e36b7b8f39d798a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42835 } }
I20260620 22:55:39.835156 20503 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:39.835186 20503 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:39.835232 20503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:39.835698 20503 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd908dc2b2847568e36b7b8f39d798a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42835 } }
I20260620 22:55:39.835784 20503 leader_election.cc:302] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd908dc2b2847568e36b7b8f39d798a; no voters:
I20260620 22:55:39.835933 20503 leader_election.cc:288] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:39.835961 20506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:39.836094 20506 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [term 1 LEADER]: Becoming Leader. State: Replica: ccd908dc2b2847568e36b7b8f39d798a, State: Running, Role: LEADER
I20260620 22:55:39.836153 20503 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:39.836180 20506 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd908dc2b2847568e36b7b8f39d798a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42835 } }
I20260620 22:55:39.836395 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20435
I20260620 22:55:39.836509 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:39.836498 20507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd908dc2b2847568e36b7b8f39d798a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd908dc2b2847568e36b7b8f39d798a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42835 } } }
I20260620 22:55:39.836567 20507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:39.836648 20508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd908dc2b2847568e36b7b8f39d798a. Latest consensus state: current_term: 1 leader_uuid: "ccd908dc2b2847568e36b7b8f39d798a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd908dc2b2847568e36b7b8f39d798a" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 42835 } } }
I20260620 22:55:39.836719 20508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:39.836778 20510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:39.836874 2410 external_mini_cluster.cc:1428] Setting key b653de66356552eea7e5d551201b14d7
I20260620 22:55:39.837312 20510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:39.838690 20510 catalog_manager.cc:1347] Generated new cluster ID: 7d259a81ebb5490d825e2653c92d8ae9
I20260620 22:55:39.838738 20510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:55:39 dist-test-slave-4mtw krb5kdc[20422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:39.848070 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:39.837431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.2.90.190:42835)
0620 22:55:39.837636 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:39.837645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:39.837872 (+ 227us) server_negotiation.cc:407] Connection header received
0620 22:55:39.838060 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:39.838067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:39.838125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:39.838241 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:39.839050 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:39.840435 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:39.841509 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:39.842067 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:39.842163 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:39.844759 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:39.844763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:39.844763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:39.844775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:39.846567 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:39.846908 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:39.846911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:39.846911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:39.847015 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:39.847180 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:39.847181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:39.847181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:39.847471 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:39.847618 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:39.847708 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 22:55:39.847837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":159,"thread_start_us":62,"threads_started":1}
I20260620 22:55:39.849064 20510 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:39.849534 20510 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:39.856122 20510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a: Generated new TSK 0
I20260620 22:55:39.856261 20510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:55:39.879042 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42835
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:39.957754 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:39.957966 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:39.957998 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:39.959124 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:39.959167 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:39.959201 20529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:39.960618 20529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:39.960873 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:39.961076 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:39.965638 20534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:39.965653 20535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:39.965674 20537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:39.965822 20529 server_base.cc:1034] running on GCE node
I20260620 22:55:39.965988 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:39.966199 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:39.967350 20529 hybrid_clock.cc:648] HybridClock initialized: now 1781996139967339 us; error 28 us; skew 500 ppm
Jun 20 22:55:39 dist-test-slave-4mtw krb5kdc[20422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996139, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:39.969702 20529 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:55:39.970089 20529 webserver.cc:460] Webserver started at http://127.2.90.129:42497/ using document root <none> and password file <none>
I20260620 22:55:39.970292 20529 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:39.970351 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:39.970474 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:39.971494 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "86bd6efbec93449c8c3b9a554a6e377c"
format_stamp: "Formatted at 2026-06-20 22:55:39 on dist-test-slave-4mtw"
server_key: "113e9147c3acf84b0b9d2aa047a82db7"
server_key_iv: "11be0ad8445d11299c0666717102b550"
server_key_version: "encryptionkey@0"
I20260620 22:55:39.971807 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "86bd6efbec93449c8c3b9a554a6e377c"
format_stamp: "Formatted at 2026-06-20 22:55:39 on dist-test-slave-4mtw"
server_key: "113e9147c3acf84b0b9d2aa047a82db7"
server_key_iv: "11be0ad8445d11299c0666717102b550"
server_key_version: "encryptionkey@0"
I20260620 22:55:39.973065 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 22:55:39.973682 20544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:39.973826 20529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:39.973905 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "86bd6efbec93449c8c3b9a554a6e377c"
format_stamp: "Formatted at 2026-06-20 22:55:39 on dist-test-slave-4mtw"
server_key: "113e9147c3acf84b0b9d2aa047a82db7"
server_key_iv: "11be0ad8445d11299c0666717102b550"
server_key_version: "encryptionkey@0"
I20260620 22:55:39.973964 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:39.988725 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:39.990769 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:39.990903 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:39.991107 20529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:39.991387 20529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:39.991420 20529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:39.991447 20529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:39.991472 20529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:39.997023 20529 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:34969
I20260620 22:55:39.997095 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:34969 every 8 connection(s)
I20260620 22:55:39.997375 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
Jun 20 22:55:39 dist-test-slave-4mtw krb5kdc[20422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996139, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:40.003528 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20529
I20260620 22:55:40.003628 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:55:40.003809 2410 external_mini_cluster.cc:1428] Setting key 3b14bb6de986d26121b7008a6d82079d
I20260620 22:55:40.006158 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:39.998027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:49147 (local address 127.2.90.190:42835)
0620 22:55:39.998083 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:39.998085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:39.998274 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:55:39.998737 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:39.998738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:39.998746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:39.998800 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:39.999671 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.000162 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.001148 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.001377 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.001443 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.003405 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.003406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.003407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:40.003413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.004811 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.005250 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.005251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.005251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.005297 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.005531 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.005531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.005532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.005623 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.005665 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.005982 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.006066 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:55:40.006263 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:39.998003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42835 (local address 127.2.90.129:49147)
0620 22:55:39.998157 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:39.998198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:39.998663 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:39.998856 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:39.998859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:39.999400 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:39.999594 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:39.999599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.000224 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.000224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.001055 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.001057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.001449 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.001449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.001550 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.001983 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:40.001991 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:40.003305 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:40.004886 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:40.004895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:40.004898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:40.005187 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:40.005351 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:40.005352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:40.005352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:40.005480 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:40.005669 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:40.005673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:40.005840 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.005953 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.006083 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 22:55:40.006713 20658 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42835
I20260620 22:55:40.006805 20658 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:40.006994 20658 heartbeater.cc:511] Master 127.2.90.190:42835 requested a full tablet report, sending...
I20260620 22:55:40.007490 20467 ts_manager.cc:194] Registered new tserver with Master: 86bd6efbec93449c8c3b9a554a6e377c (127.2.90.129:34969)
I20260620 22:55:40.008199 20467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:49147
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:55:40.025405 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42835
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:40.102991 20663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:40.103181 20663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:40.103210 20663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:40.104305 20663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:40.104343 20663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:40.104377 20663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:40.105770 20663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:40.105994 20663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:40.106189 20663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:40.110700 20668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.110780 20669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.110963 20663 server_base.cc:1034] running on GCE node
W20260620 22:55:40.110987 20671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.111148 20663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:40.111370 20663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:40.112510 20663 hybrid_clock.cc:648] HybridClock initialized: now 1781996140112497 us; error 30 us; skew 500 ppm
Jun 20 22:55:40 dist-test-slave-4mtw krb5kdc[20422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:40.114815 20663 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:55:40.115249 20663 webserver.cc:460] Webserver started at http://127.2.90.130:37991/ using document root <none> and password file <none>
I20260620 22:55:40.115449 20663 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:40.115496 20663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:40.115620 20663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:40.116636 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "b41430f23cfe4c5e87b5a74fce7b98b0"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "584445897d89e4d8e02935f106a43c2e"
server_key_iv: "80d60141628183ccf66fdb1e6d969693"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.117024 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "b41430f23cfe4c5e87b5a74fce7b98b0"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "584445897d89e4d8e02935f106a43c2e"
server_key_iv: "80d60141628183ccf66fdb1e6d969693"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.118211 20663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:40.118893 20678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.119052 20663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.119122 20663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "b41430f23cfe4c5e87b5a74fce7b98b0"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "584445897d89e4d8e02935f106a43c2e"
server_key_iv: "80d60141628183ccf66fdb1e6d969693"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.119186 20663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:40.139199 20663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:40.141290 20663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:40.141425 20663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:40.141633 20663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:40.141917 20663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:40.141950 20663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.141971 20663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:40.141995 20663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.147323 20663 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:33089
I20260620 22:55:40.147388 20791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:33089 every 8 connection(s)
I20260620 22:55:40.147694 20663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:40.149389 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20663
I20260620 22:55:40.149474 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:55:40.149657 2410 external_mini_cluster.cc:1428] Setting key 726e6fa357a3cef2ca031fdb2c8e1604
Jun 20 22:55:40 dist-test-slave-4mtw krb5kdc[20422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
I20260620 22:55:40.156752 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.148347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:48103 (local address 127.2.90.190:42835)
0620 22:55:40.148398 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.148400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.148588 (+ 188us) server_negotiation.cc:407] Connection header received
0620 22:55:40.149130 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.149131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.149139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.149194 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:40.150240 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.150911 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.152072 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.152270 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.152321 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.154315 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.154316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.154317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:40.154323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.155549 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.155897 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.155898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.155899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.155950 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.156155 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.156156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.156157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.156258 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.156310 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.156552 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.156630 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:55:40.156908 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.148310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42835 (local address 127.2.90.130:48103)
0620 22:55:40.148469 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.148510 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.149048 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.149275 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.149279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.149967 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:40.150162 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.150167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.150988 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.150989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.151993 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.151996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.152346 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.152347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.152481 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.152980 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:40.152988 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:40.154230 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:40.155619 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:40.155626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:40.155628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:40.155834 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:40.156002 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:40.156003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:40.156003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:40.156104 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:40.156309 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:40.156315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:40.156440 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.156525 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.156671 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 22:55:40.157302 20792 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42835
I20260620 22:55:40.157384 20792 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:40.157562 20792 heartbeater.cc:511] Master 127.2.90.190:42835 requested a full tablet report, sending...
I20260620 22:55:40.157894 20467 ts_manager.cc:194] Registered new tserver with Master: b41430f23cfe4c5e87b5a74fce7b98b0 (127.2.90.130:33089)
I20260620 22:55:40.158237 20467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:48103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:55:40.172009 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42835
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:40.249935 20797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:40.250113 20797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:40.250133 20797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:40.251224 20797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:40.251263 20797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:40.251293 20797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:40.252739 20797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:42835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:40.252954 20797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:40.253150 20797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:40.257717 20803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.257807 20797 server_base.cc:1034] running on GCE node
W20260620 22:55:40.257731 20802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.257723 20805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.258100 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:40.258322 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:40.259447 20797 hybrid_clock.cc:648] HybridClock initialized: now 1781996140259441 us; error 35 us; skew 500 ppm
Jun 20 22:55:40 dist-test-slave-4mtw krb5kdc[20422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:40.261803 20797 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:55:40.262217 20797 webserver.cc:460] Webserver started at http://127.2.90.131:35349/ using document root <none> and password file <none>
I20260620 22:55:40.262389 20797 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:40.262427 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:40.262523 20797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:40.263476 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "c93a66da0f5e4608b7fc9fba21971de2"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "7edd43f1164eae78ca7dd45898ae7b27"
server_key_iv: "102810769bed2e910e7cc8439afeade6"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.263753 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "c93a66da0f5e4608b7fc9fba21971de2"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "7edd43f1164eae78ca7dd45898ae7b27"
server_key_iv: "102810769bed2e910e7cc8439afeade6"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.264950 20797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:40.265581 20812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.265750 20797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.265834 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "c93a66da0f5e4608b7fc9fba21971de2"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "7edd43f1164eae78ca7dd45898ae7b27"
server_key_iv: "102810769bed2e910e7cc8439afeade6"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.265888 20797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:40.285698 20797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:40.287381 20797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:40.287504 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:40.287703 20797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:40.287977 20797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:40.288007 20797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.288029 20797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:40.288053 20797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.293237 20797 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:34449
I20260620 22:55:40.293316 20925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:34449 every 8 connection(s)
I20260620 22:55:40.293566 20797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:40.296063 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20797
I20260620 22:55:40.296173 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:55:40.296339 2410 external_mini_cluster.cc:1428] Setting key 54f769db3c648452e057fe72b284510d
Jun 20 22:55:40 dist-test-slave-4mtw krb5kdc[20422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:40.302160 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.294180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:33427 (local address 127.2.90.190:42835)
0620 22:55:40.294252 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.294254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.294426 (+ 172us) server_negotiation.cc:407] Connection header received
0620 22:55:40.294876 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.294877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.294886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.294935 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:40.295780 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.296496 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.297597 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.297792 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.297844 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.299756 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.299758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.299758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:40.299764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.300987 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.301326 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.301327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.301327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.301387 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.301589 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.301590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.301590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.301700 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.301752 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.302001 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.302074 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 22:55:40.302311 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.294151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:42835 (local address 127.2.90.131:33427)
0620 22:55:40.294305 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.294353 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.294803 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.294987 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.294990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.295535 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:40.295718 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.295723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.296582 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.296583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.297515 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.297517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.297856 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.297857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.297981 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.298420 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:40.298427 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:40.299674 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:40.301055 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:40.301061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:40.301063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:40.301264 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:40.301438 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:40.301439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:40.301439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:40.301537 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:40.301749 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:40.301756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:40.301888 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.301972 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.302114 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 22:55:40.302666 20926 heartbeater.cc:347] Connected to a master server at 127.2.90.190:42835
I20260620 22:55:40.302747 20926 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:40.302909 20926 heartbeater.cc:511] Master 127.2.90.190:42835 requested a full tablet report, sending...
I20260620 22:55:40.303216 20467 ts_manager.cc:194] Registered new tserver with Master: c93a66da0f5e4608b7fc9fba21971de2 (127.2.90.131:34449)
I20260620 22:55:40.303561 20467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:33427
I20260620 22:55:40.307876 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:40.314229 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.309118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.2.90.190:42835)
0620 22:55:40.309168 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.309169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.309262 (+ 93us) server_negotiation.cc:407] Connection header received
0620 22:55:40.309307 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.309308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.309314 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.309361 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:40.309814 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.310260 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.310961 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.311139 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.311187 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.312296 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.312297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.312297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:40.312301 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.313370 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.313692 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.313693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.313693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.313737 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:40.313875 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:40.313875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:40.313875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:40.313978 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.314048 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.314128 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.314170 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:55:40.315601 20467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:40.317162 20467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:40.322916 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.319363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.2.90.131:34449)
0620 22:55:40.319618 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.319622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.319630 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:40.319663 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.319665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.319679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.319893 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:40.319956 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.320928 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.321913 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.322508 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.322576 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.322636 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.322787 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.322829 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":69,"threads_started":1}
I20260620 22:55:40.322943 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.319253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:34449 (local address 127.0.0.1:41936)
0620 22:55:40.319462 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.319471 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.319514 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.319769 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.319769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.319771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:40.319817 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.319818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.321004 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.321005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.321832 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.321835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.322592 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.322593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.322668 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.322683 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.322779 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.322859 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260620 22:55:40.323077 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.318916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:34969 (local address 127.0.0.1:53304)
0620 22:55:40.319031 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.319059 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.319111 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.319407 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.319409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.319412 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:40.319476 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.319477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.320779 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.320780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.322148 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.322150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.322901 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.322902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.322958 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.322961 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.322988 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.323020 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":53,"thread_start_us":80,"threads_started":1}
I20260620 22:55:40.323436 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.319024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.2.90.129:34969)
0620 22:55:40.319184 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.319189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.319198 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:40.319232 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.319234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.319249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.319318 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:40.319546 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.320640 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.322239 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.322812 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.323224 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.323290 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.323316 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.323354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260620 22:55:40.323598 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.319133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:33089 (local address 127.0.0.1:42154)
0620 22:55:40.319205 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.319213 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.319256 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.319830 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.319831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.319833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:40.319880 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.319881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.320774 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.320775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.322147 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.322150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.323322 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.323323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.323370 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.323373 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.323418 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.323459 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":53,"thread_start_us":41,"threads_started":1}
I20260620 22:55:40.323642 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.319460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.2.90.130:33089)
0620 22:55:40.319616 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.319620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.319629 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:40.319663 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.319665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.319679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.319751 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:40.319956 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.320655 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.322229 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.323179 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.323241 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.323295 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.323443 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.323481 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260620 22:55:40.323695 20860 tablet_service.cc:1467] Processing CreateTablet for tablet fb07b42e841e4473a009e664186ffa56 (DEFAULT_TABLE table=test-table [id=50f4f81d31d54b9299e74b7044946aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:40.324043 20860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb07b42e841e4473a009e664186ffa56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:40.323975 20592 tablet_service.cc:1467] Processing CreateTablet for tablet fb07b42e841e4473a009e664186ffa56 (DEFAULT_TABLE table=test-table [id=50f4f81d31d54b9299e74b7044946aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:40.324249 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb07b42e841e4473a009e664186ffa56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:40.324311 20726 tablet_service.cc:1467] Processing CreateTablet for tablet fb07b42e841e4473a009e664186ffa56 (DEFAULT_TABLE table=test-table [id=50f4f81d31d54b9299e74b7044946aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:40.324640 20726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb07b42e841e4473a009e664186ffa56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:40.327291 20940 tablet_bootstrap.cc:492] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Bootstrap starting.
I20260620 22:55:40.327503 20941 tablet_bootstrap.cc:492] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c: Bootstrap starting.
I20260620 22:55:40.327898 20940 tablet_bootstrap.cc:654] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:40.327988 20941 tablet_bootstrap.cc:654] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:40.328349 20940 log.cc:824] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:40.328415 20941 log.cc:824] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:40.328661 20942 tablet_bootstrap.cc:492] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Bootstrap starting.
I20260620 22:55:40.329092 20941 tablet_bootstrap.cc:492] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c: No bootstrap required, opened a new log
I20260620 22:55:40.329092 20940 tablet_bootstrap.cc:492] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: No bootstrap required, opened a new log
I20260620 22:55:40.329161 20940 ts_tablet_manager.cc:1397] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:40.329161 20941 ts_tablet_manager.cc:1397] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:40.329226 20942 tablet_bootstrap.cc:654] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:40.329658 20942 log.cc:824] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:40.330247 20942 tablet_bootstrap.cc:492] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: No bootstrap required, opened a new log
I20260620 22:55:40.330303 20942 ts_tablet_manager.cc:1397] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 22:55:40.331032 20941 raft_consensus.cc:348] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.331032 20940 raft_consensus.cc:348] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.331213 20941 raft_consensus.cc:374] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:40.331213 20940 raft_consensus.cc:374] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:40.331250 20940 raft_consensus.cc:729] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41430f23cfe4c5e87b5a74fce7b98b0, State: Initialized, Role: FOLLOWER
I20260620 22:55:40.331250 20941 raft_consensus.cc:729] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bd6efbec93449c8c3b9a554a6e377c, State: Initialized, Role: FOLLOWER
I20260620 22:55:40.331387 20941 consensus_queue.cc:260] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.331387 20940 consensus_queue.cc:260] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.331657 20941 ts_tablet_manager.cc:1428] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:55:40.331657 20940 ts_tablet_manager.cc:1428] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 22:55:40.331681 20792 heartbeater.cc:503] Master 127.2.90.190:42835 was elected leader, sending a full tablet report...
I20260620 22:55:40.331780 20658 heartbeater.cc:503] Master 127.2.90.190:42835 was elected leader, sending a full tablet report...
I20260620 22:55:40.332163 20942 raft_consensus.cc:348] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.332307 20942 raft_consensus.cc:374] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:40.332340 20942 raft_consensus.cc:729] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93a66da0f5e4608b7fc9fba21971de2, State: Initialized, Role: FOLLOWER
I20260620 22:55:40.332453 20942 consensus_queue.cc:260] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.332723 20942 ts_tablet_manager.cc:1428] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:55:40.332742 20926 heartbeater.cc:503] Master 127.2.90.190:42835 was elected leader, sending a full tablet report...
I20260620 22:55:40.387795 20947 raft_consensus.cc:482] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:40.387916 20947 raft_consensus.cc:504] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.388273 20947 leader_election.cc:288] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b41430f23cfe4c5e87b5a74fce7b98b0 (127.2.90.130:33089), c93a66da0f5e4608b7fc9fba21971de2 (127.2.90.131:34449)
I20260620 22:55:40.390842 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.388428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:33089 (local address 127.2.90.129:33133)
0620 22:55:40.388480 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.388491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.388543 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.388704 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.388705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.388707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:40.388772 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.388774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.389359 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.389370 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.390087 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.390089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.390606 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.390607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.390677 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.390680 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.390730 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.390771 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260620 22:55:40.390849 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.388511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:33133 (local address 127.2.90.130:33089)
0620 22:55:40.388559 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.388562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.388570 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:40.388598 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.388599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.388612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.388663 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:40.388834 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.389288 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.390172 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.390536 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.390597 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.390605 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.390740 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.390784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:55:40.391067 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.388628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:55937 (local address 127.2.90.131:34449)
0620 22:55:40.388671 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.388672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.388769 (+ 97us) server_negotiation.cc:407] Connection header received
0620 22:55:40.388811 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.388811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.388819 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.388873 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:40.389010 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.389450 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.390327 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.390705 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.390765 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.390772 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.390963 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.391003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:55:40.391220 20746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb07b42e841e4473a009e664186ffa56" candidate_uuid: "86bd6efbec93449c8c3b9a554a6e377c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" is_pre_election: true
I20260620 22:55:40.391283 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.388525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:34449 (local address 127.2.90.129:55937)
0620 22:55:40.388708 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.388717 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.388764 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.388881 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.388881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.388882 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:40.388953 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.388955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.389515 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.389516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.390244 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.390246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.390803 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.390804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.390880 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.390884 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.391177 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.391219 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
I20260620 22:55:40.391379 20746 raft_consensus.cc:2454] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bd6efbec93449c8c3b9a554a6e377c in term 0.
I20260620 22:55:40.391572 20547 leader_election.cc:302] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bd6efbec93449c8c3b9a554a6e377c, b41430f23cfe4c5e87b5a74fce7b98b0; no voters:
I20260620 22:55:40.391623 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb07b42e841e4473a009e664186ffa56" candidate_uuid: "86bd6efbec93449c8c3b9a554a6e377c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93a66da0f5e4608b7fc9fba21971de2" is_pre_election: true
I20260620 22:55:40.391700 20947 raft_consensus.cc:2781] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:40.391762 20947 raft_consensus.cc:482] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:40.391762 20880 raft_consensus.cc:2454] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bd6efbec93449c8c3b9a554a6e377c in term 0.
I20260620 22:55:40.391791 20947 raft_consensus.cc:3037] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:40.392506 20947 raft_consensus.cc:504] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.392601 20947 leader_election.cc:288] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [CANDIDATE]: Term 1 election: Requested vote from peers b41430f23cfe4c5e87b5a74fce7b98b0 (127.2.90.130:33089), c93a66da0f5e4608b7fc9fba21971de2 (127.2.90.131:34449)
I20260620 22:55:40.392778 20746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb07b42e841e4473a009e664186ffa56" candidate_uuid: "86bd6efbec93449c8c3b9a554a6e377c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0"
I20260620 22:55:40.392781 20880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb07b42e841e4473a009e664186ffa56" candidate_uuid: "86bd6efbec93449c8c3b9a554a6e377c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c93a66da0f5e4608b7fc9fba21971de2"
I20260620 22:55:40.392849 20880 raft_consensus.cc:3037] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:40.392853 20746 raft_consensus.cc:3037] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:40.393517 20746 raft_consensus.cc:2454] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bd6efbec93449c8c3b9a554a6e377c in term 1.
I20260620 22:55:40.393523 20880 raft_consensus.cc:2454] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bd6efbec93449c8c3b9a554a6e377c in term 1.
I20260620 22:55:40.393664 20547 leader_election.cc:302] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bd6efbec93449c8c3b9a554a6e377c, b41430f23cfe4c5e87b5a74fce7b98b0; no voters:
I20260620 22:55:40.393752 20947 raft_consensus.cc:2781] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:40.393877 20947 raft_consensus.cc:686] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [term 1 LEADER]: Becoming Leader. State: Replica: 86bd6efbec93449c8c3b9a554a6e377c, State: Running, Role: LEADER
I20260620 22:55:40.393950 20947 consensus_queue.cc:237] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } }
I20260620 22:55:40.394680 20467 catalog_manager.cc:5495] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bd6efbec93449c8c3b9a554a6e377c (127.2.90.129). New cstate: current_term: 1 leader_uuid: "86bd6efbec93449c8c3b9a554a6e377c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bd6efbec93449c8c3b9a554a6e377c" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 34969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 } health_report { overall_health: UNKNOWN } } }
W20260620 22:55:40.398161 20793 tablet.cc:2369] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:40.413581 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.411219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.2.90.129:34969)
0620 22:55:40.411300 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.411302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.411311 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:40.411351 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.411352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.411377 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.411441 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:40.411573 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.412046 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.412820 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.413023 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.413077 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.413224 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:40.413323 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:40.413410 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.413464 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.413474 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 22:55:40.415289 20746 raft_consensus.cc:1261] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 1 FOLLOWER]: Refusing update from remote peer 86bd6efbec93449c8c3b9a554a6e377c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:40.415297 20880 raft_consensus.cc:1261] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 1 FOLLOWER]: Refusing update from remote peer 86bd6efbec93449c8c3b9a554a6e377c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:40.415616 20950 consensus_queue.cc:1035] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c93a66da0f5e4608b7fc9fba21971de2" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:40.415745 20947 consensus_queue.cc:1035] T fb07b42e841e4473a009e664186ffa56 P 86bd6efbec93449c8c3b9a554a6e377c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41430f23cfe4c5e87b5a74fce7b98b0" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:40.417862 20954 mvcc.cc:205] Tried to move back new op lower bound from 7299056191138414592 to 7299056191054143488. Current Snapshot: MvccSnapshot[applied={T|T < 7299056191138414592}]
I20260620 22:55:40.420002 20467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:40.420084 20467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:40.420794 20467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccd908dc2b2847568e36b7b8f39d798a: Sending DeleteTablet for 3 replicas of tablet fb07b42e841e4473a009e664186ffa56
I20260620 22:55:40.421126 20726 tablet_service.cc:1514] Processing DeleteTablet for tablet fb07b42e841e4473a009e664186ffa56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:40 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:42154
I20260620 22:55:40.421156 20860 tablet_service.cc:1514] Processing DeleteTablet for tablet fb07b42e841e4473a009e664186ffa56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:40 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:41936
I20260620 22:55:40.421411 20963 tablet_replica.cc:330] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: stopping tablet replica
I20260620 22:55:40.421435 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20529
I20260620 22:55:40.425235 20963 raft_consensus.cc:2229] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:40.425372 20963 raft_consensus.cc:2258] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:40.425529 20962 tablet_replica.cc:330] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: stopping tablet replica
I20260620 22:55:40.425628 20962 raft_consensus.cc:2229] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:40.425724 20962 raft_consensus.cc:2258] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:40.425881 20963 ts_tablet_manager.cc:1905] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 22:55:40.425904 20453 connection.cc:535] client connection to 127.2.90.129:34969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 22:55:40.425977 20453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 22:55:40.426221 20962 ts_tablet_manager.cc:1905] T fb07b42e841e4473a009e664186ffa56 P b41430f23cfe4c5e87b5a74fce7b98b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:40.426527 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.426419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:34969 (local address 127.0.0.1:53328)
0620 22:55:40.426461 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.426494 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:34969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:55:40.426697 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20663
W20260620 22:55:40.426697 20453 catalog_manager.cc:4584] TS 86bd6efbec93449c8c3b9a554a6e377c (127.2.90.129:34969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb07b42e841e4473a009e664186ffa56: Network error: Client connection negotiation failed: client connection to 127.2.90.129:34969: connect: Connection refused (error 111)
I20260620 22:55:40.427469 20963 ts_tablet_manager.cc:1918] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:40.427531 20963 log.cc:1196] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/fb07b42e841e4473a009e664186ffa56
I20260620 22:55:40.427745 20963 ts_tablet_manager.cc:1939] T fb07b42e841e4473a009e664186ffa56 P c93a66da0f5e4608b7fc9fba21971de2: Deleting consensus metadata
I20260620 22:55:40.428092 20453 catalog_manager.cc:4841] TS c93a66da0f5e4608b7fc9fba21971de2 (127.2.90.131:34449): tablet fb07b42e841e4473a009e664186ffa56 (table test-table [id=50f4f81d31d54b9299e74b7044946aaa]) successfully deleted
I20260620 22:55:40.431165 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20797
I20260620 22:55:40.431187 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.431095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:33089 (local address 127.0.0.1:42160)
0620 22:55:40.431138 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.431163 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:33089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 22:55:40.435405 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20435
2026-06-20T22:55:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1768 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T22:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:40Z Disabled control of system clock
I20260620 22:55:40.459841 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:34165
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:40.536028 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:40.536224 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:40.536243 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:40.537336 20968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:40.537370 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:40.537381 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:40.537392 20968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:40.537403 20968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:40.538638 20968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:34165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:40.538815 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:40.539032 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:40.543434 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.543506 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.543440 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.543581 20968 server_base.cc:1034] running on GCE node
I20260620 22:55:40.543788 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:40.544046 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:40.545182 20968 hybrid_clock.cc:648] HybridClock initialized: now 1781996140545161 us; error 36 us; skew 500 ppm
I20260620 22:55:40.546307 20968 webserver.cc:460] Webserver started at http://127.2.90.190:37199/ using document root <none> and password file <none>
I20260620 22:55:40.546514 20968 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:40.546561 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:40.546692 20968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:40.547676 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "90d785eed8214d29b7a57fe7a24bb1b9"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "ae011b5e8ff2f21fd8ff65ea4d031c6f"
server_key_iv: "37294d74db35083575bb031128d1146e"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.547995 20968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "90d785eed8214d29b7a57fe7a24bb1b9"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "ae011b5e8ff2f21fd8ff65ea4d031c6f"
server_key_iv: "37294d74db35083575bb031128d1146e"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.549214 20968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:40.549847 20982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.549989 20968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:40.550068 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "90d785eed8214d29b7a57fe7a24bb1b9"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "ae011b5e8ff2f21fd8ff65ea4d031c6f"
server_key_iv: "37294d74db35083575bb031128d1146e"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.550127 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:40.565968 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:40.566255 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:40.566383 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:40.570168 20968 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:34165
I20260620 22:55:40.570202 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:34165 every 8 connection(s)
I20260620 22:55:40.570523 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:40.571409 21035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:40.573506 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 20968
I20260620 22:55:40.573604 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:40.573720 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9: Bootstrap starting.
I20260620 22:55:40.573764 2410 external_mini_cluster.cc:1428] Setting key 842b3174a5d8d835f2d54fc067293645
I20260620 22:55:40.574399 21035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:40.574702 21035 log.cc:824] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:40.575444 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9: No bootstrap required, opened a new log
I20260620 22:55:40.577606 21035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34165 } }
I20260620 22:55:40.577769 21035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:40.577800 21035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d785eed8214d29b7a57fe7a24bb1b9, State: Initialized, Role: FOLLOWER
I20260620 22:55:40.577925 21035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34165 } }
I20260620 22:55:40.577994 21035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:40.578028 21035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:40.578070 21035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:40.578663 21035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34165 } }
I20260620 22:55:40.578749 21035 leader_election.cc:302] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90d785eed8214d29b7a57fe7a24bb1b9; no voters:
I20260620 22:55:40.578907 21035 leader_election.cc:288] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:40.578963 21040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:40.579098 21040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [term 1 LEADER]: Becoming Leader. State: Replica: 90d785eed8214d29b7a57fe7a24bb1b9, State: Running, Role: LEADER
I20260620 22:55:40.579154 21035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:40.579185 21040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34165 } }
I20260620 22:55:40.579509 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90d785eed8214d29b7a57fe7a24bb1b9. Latest consensus state: current_term: 1 leader_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34165 } } }
I20260620 22:55:40.579586 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:40.579741 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d785eed8214d29b7a57fe7a24bb1b9" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34165 } } }
I20260620 22:55:40.579797 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:40.579818 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.574235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.2.90.190:34165)
0620 22:55:40.574453 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.574462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.574497 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:55:40.575037 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.575044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.575106 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.575250 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:40.575457 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.576846 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.577574 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.578150 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.578274 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.578843 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.578847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.578847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:40.579140 (+ 293us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.579228 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:40.579246 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.579506 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.579539 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.579623 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260620 22:55:40.579910 21047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:40.580353 21047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:40.581787 21047 catalog_manager.cc:1347] Generated new cluster ID: 34d17166802c421fa52ac27ca70f8aaf
I20260620 22:55:40.581838 21047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:40.616529 21047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:40.617201 21047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:40.626299 21047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9: Generated new TSK 0
I20260620 22:55:40.626457 21047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:40.639078 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:34165
--builtin_ntp_servers=127.2.90.148:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:40.716725 21059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:40.716921 21059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:40.716940 21059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:40.718060 21059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:40.718096 21059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:40.718122 21059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:40.719446 21059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:40.719655 21059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:40.719915 21059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:40.724514 21065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.724509 21067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.724709 21059 server_base.cc:1034] running on GCE node
W20260620 22:55:40.724510 21064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.724939 21059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:40.725163 21059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:40.726296 21059 hybrid_clock.cc:648] HybridClock initialized: now 1781996140726282 us; error 26 us; skew 500 ppm
I20260620 22:55:40.727433 21059 webserver.cc:460] Webserver started at http://127.2.90.129:41177/ using document root <none> and password file <none>
I20260620 22:55:40.727644 21059 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:40.727691 21059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:40.727814 21059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:40.728873 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "8f2f0221028b4293827356b5320e749d"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "745184396c66565c2a3bcd26afa43cbd"
server_key_iv: "0f6230e10041e540969e92cc2d39733e"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.729187 21059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "8f2f0221028b4293827356b5320e749d"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "745184396c66565c2a3bcd26afa43cbd"
server_key_iv: "0f6230e10041e540969e92cc2d39733e"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.730317 21059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:40.730904 21073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.731045 21059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:40.731122 21059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "8f2f0221028b4293827356b5320e749d"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "745184396c66565c2a3bcd26afa43cbd"
server_key_iv: "0f6230e10041e540969e92cc2d39733e"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.731179 21059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:40.741046 21059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:40.741284 21059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:40.741406 21059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:40.741585 21059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:40.741868 21059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:40.741905 21059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.741931 21059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:40.741954 21059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.747327 21059 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:46767
I20260620 22:55:40.747390 21186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:46767 every 8 connection(s)
I20260620 22:55:40.747658 21059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:55:40.752126 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.748289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:33139 (local address 127.2.90.190:34165)
0620 22:55:40.748366 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.748367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.748560 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:55:40.749058 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.749060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.749069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.749104 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:40.749699 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.750175 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.751062 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.751231 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.751271 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.751958 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.751959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.751959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:40.751962 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.751971 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:40.751974 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.752006 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.752057 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.752073 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:55:40.752334 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.748283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:34165 (local address 127.2.90.129:33139)
0620 22:55:40.748453 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.748491 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.748965 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.749163 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.749166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.749173 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:40.749642 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.749646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.750233 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.750233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.751004 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.751006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.751281 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.751282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.751389 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.751851 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:40.751851 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:40.751880 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:40.751901 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:40.752008 (+ 107us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:40.752010 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:40.752012 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.752055 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.752126 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260620 22:55:40.752799 21187 heartbeater.cc:347] Connected to a master server at 127.2.90.190:34165
I20260620 22:55:40.752884 21187 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:40.753018 21187 heartbeater.cc:511] Master 127.2.90.190:34165 requested a full tablet report, sending...
I20260620 22:55:40.753446 20999 ts_manager.cc:194] Registered new tserver with Master: 8f2f0221028b4293827356b5320e749d (127.2.90.129:46767)
I20260620 22:55:40.753537 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21059
I20260620 22:55:40.753669 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:55:40.753844 2410 external_mini_cluster.cc:1428] Setting key 5e7bae13464c7c760011e70c858e1697
I20260620 22:55:40.754319 20999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.129:33139
I20260620 22:55:40.754781 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:34165
--builtin_ntp_servers=127.2.90.148:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:40.831419 21190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:40.831612 21190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:40.831640 21190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:40.832788 21190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:40.832826 21190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:40.832859 21190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:40.834183 21190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:40.834421 21190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:40.834709 21190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:40.839257 21196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.839308 21198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.839232 21195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.839419 21190 server_base.cc:1034] running on GCE node
I20260620 22:55:40.839612 21190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:40.839798 21190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:40.840929 21190 hybrid_clock.cc:648] HybridClock initialized: now 1781996140840913 us; error 33 us; skew 500 ppm
I20260620 22:55:40.842001 21190 webserver.cc:460] Webserver started at http://127.2.90.130:37501/ using document root <none> and password file <none>
I20260620 22:55:40.842201 21190 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:40.842243 21190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:40.842372 21190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:40.843307 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "c1c9efb2d3ea4befbac2341890a34764"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "6550aaa6393e3c5d130e3264521142df"
server_key_iv: "1eee92eeddac69c863a92c4e0ef8b4d9"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.843614 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "c1c9efb2d3ea4befbac2341890a34764"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "6550aaa6393e3c5d130e3264521142df"
server_key_iv: "1eee92eeddac69c863a92c4e0ef8b4d9"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.844738 21190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 22:55:40.845374 21204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.845513 21190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:40.845587 21190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "c1c9efb2d3ea4befbac2341890a34764"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "6550aaa6393e3c5d130e3264521142df"
server_key_iv: "1eee92eeddac69c863a92c4e0ef8b4d9"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.845649 21190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:40.864774 21190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:40.865034 21190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:40.865154 21190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:40.865355 21190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:40.865648 21190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:40.865679 21190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.865705 21190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:40.865726 21190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.870844 21190 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:42797
I20260620 22:55:40.870906 21317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:42797 every 8 connection(s)
I20260620 22:55:40.871181 21190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:40.875785 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.871825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:50203 (local address 127.2.90.190:34165)
0620 22:55:40.871885 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:40.871887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:40.872120 (+ 233us) server_negotiation.cc:407] Connection header received
0620 22:55:40.872621 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:40.872622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:40.872632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:40.872710 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:40.873345 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.873825 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.874702 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.874881 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.874922 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.875592 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:40.875593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:40.875594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:40.875598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:40.875612 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:40.875616 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:40.875652 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:40.875705 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 22:55:40.875726 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 22:55:40.875979 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:40.871835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:34165 (local address 127.2.90.130:50203)
0620 22:55:40.872010 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:40.872042 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:40.872517 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:40.872782 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:40.872785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:40.872794 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:40.873280 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.873284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.873888 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.873888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.874644 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:40.874646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:40.874934 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:40.874934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:40.875036 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:40.875475 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:40.875476 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:40.875503 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:40.875524 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:40.875653 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:40.875656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:40.875658 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:40.875700 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:55:40.875773 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260620 22:55:40.876428 21318 heartbeater.cc:347] Connected to a master server at 127.2.90.190:34165
I20260620 22:55:40.876515 21318 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:40.876662 21318 heartbeater.cc:511] Master 127.2.90.190:34165 requested a full tablet report, sending...
I20260620 22:55:40.877085 20999 ts_manager.cc:194] Registered new tserver with Master: c1c9efb2d3ea4befbac2341890a34764 (127.2.90.130:42797)
I20260620 22:55:40.877449 20999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.130:50203
I20260620 22:55:40.878702 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21190
I20260620 22:55:40.878803 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:55:40.878978 2410 external_mini_cluster.cc:1428] Setting key 4f7a808c131416773924184e783b68f5
I20260620 22:55:40.879866 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:34165
--builtin_ntp_servers=127.2.90.148:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:40.958746 21321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:40.958948 21321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:40.958967 21321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:40.960101 21321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:40.960140 21321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:40.960168 21321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:40.961519 21321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:40.961727 21321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:40.961992 21321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:40.966604 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.966688 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:40.966598 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:40.966811 21321 server_base.cc:1034] running on GCE node
I20260620 22:55:40.967010 21321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:40.967216 21321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:40.968364 21321 hybrid_clock.cc:648] HybridClock initialized: now 1781996140968347 us; error 28 us; skew 500 ppm
I20260620 22:55:40.969588 21321 webserver.cc:460] Webserver started at http://127.2.90.131:39585/ using document root <none> and password file <none>
I20260620 22:55:40.969817 21321 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:40.969866 21321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:40.969988 21321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:40.971030 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "ceb43b3178894aca87f013f7d1e615c8"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "746af69ecc28892469cee3ca5d1d0136"
server_key_iv: "d4bd0075e4ce9010ffcc7e55f07bea95"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.971349 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "ceb43b3178894aca87f013f7d1e615c8"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "746af69ecc28892469cee3ca5d1d0136"
server_key_iv: "d4bd0075e4ce9010ffcc7e55f07bea95"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.972508 21321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:40.973178 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.973318 21321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.973394 21321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "ceb43b3178894aca87f013f7d1e615c8"
format_stamp: "Formatted at 2026-06-20 22:55:40 on dist-test-slave-4mtw"
server_key: "746af69ecc28892469cee3ca5d1d0136"
server_key_iv: "d4bd0075e4ce9010ffcc7e55f07bea95"
server_key_version: "encryptionkey@0"
I20260620 22:55:40.973455 21321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:40.996025 21321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:40.996309 21321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:40.996430 21321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:40.996635 21321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:40.996968 21321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:40.997000 21321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:40.997028 21321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:40.997049 21321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:41.002295 21321 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:38459
I20260620 22:55:41.002346 21448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:38459 every 8 connection(s)
I20260620 22:55:41.002645 21321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:41.004127 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21321
I20260620 22:55:41.004232 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:55:41.004403 2410 external_mini_cluster.cc:1428] Setting key 5e40dcb4e602a30e43e4c9e077372b1c
I20260620 22:55:41.007673 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.003343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:51569 (local address 127.2.90.190:34165)
0620 22:55:41.003395 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.003397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.003615 (+ 218us) server_negotiation.cc:407] Connection header received
0620 22:55:41.004115 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.004117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.004131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.004179 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:41.004994 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.005494 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.006437 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.006625 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.006679 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.007433 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:41.007435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:41.007435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:41.007439 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:41.007454 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:41.007458 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:41.007500 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.007570 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.007596 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:55:41.007875 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.003330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:34165 (local address 127.2.90.131:51569)
0620 22:55:41.003505 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.003538 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:41.004021 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:41.004262 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:41.004266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:41.004285 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:41.004918 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.004926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.005580 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.005581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.006361 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.006363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.006693 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.006694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.006829 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.007306 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:41.007306 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:41.007336 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:41.007357 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:41.007505 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:41.007510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:41.007513 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:41.007574 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:55:41.007673 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260620 22:55:41.008266 21449 heartbeater.cc:347] Connected to a master server at 127.2.90.190:34165
I20260620 22:55:41.008354 21449 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:41.008491 21449 heartbeater.cc:511] Master 127.2.90.190:34165 requested a full tablet report, sending...
I20260620 22:55:41.008817 20999 ts_manager.cc:194] Registered new tserver with Master: ceb43b3178894aca87f013f7d1e615c8 (127.2.90.131:38459)
I20260620 22:55:41.009166 20999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.131:51569
I20260620 22:55:41.015918 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:41.019843 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.017173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.2.90.190:34165)
0620 22:55:41.017225 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.017227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.017313 (+ 86us) server_negotiation.cc:407] Connection header received
0620 22:55:41.017375 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.017376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.017384 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.017430 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:41.017574 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.018036 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.018747 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.018933 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.018984 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.019572 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:41.019573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:41.019573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:41.019577 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:41.019590 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:41.019593 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:41.019671 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.019754 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.019764 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 22:55:41.021236 20999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:41.022734 20999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:41.028478 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.024648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42797 (local address 127.0.0.1:52592)
0620 22:55:41.024775 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.024811 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:41.024870 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:41.025179 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:41.025182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:41.025184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:41.025244 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.025246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.026267 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.026268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.027324 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.027326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.028185 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.028186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.028257 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.028270 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 22:55:41.028356 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 22:55:41.028407 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260620 22:55:41.028496 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.024756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.2.90.130:42797)
0620 22:55:41.024993 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.024997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.025006 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:41.025041 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.025043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.025059 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.025169 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:41.025319 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.026197 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.027414 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.028108 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.028174 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.028258 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.028381 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.028418 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260620 22:55:41.028856 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.024867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:38459 (local address 127.0.0.1:58800)
0620 22:55:41.024990 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.024998 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:41.025033 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:41.025465 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:41.025466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:41.025468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:41.025504 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.025506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.026727 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.026728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.027766 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.027768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.028514 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.028515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.028706 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.028710 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:41.028750 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 22:55:41.028784 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":21,"thread_start_us":82,"threads_started":1}
I20260620 22:55:41.028955 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.025036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46767 (local address 127.0.0.1:36378)
0620 22:55:41.025312 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.025321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:41.025366 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:41.025618 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:41.025619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:41.025621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:41.025659 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.025660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.026885 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.026886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.027771 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.027773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.028731 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.028732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.028790 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.028795 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:55:41.028841 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 22:55:41.028889 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":60,"threads_started":1}
I20260620 22:55:41.029065 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.025157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.2.90.131:38459)
0620 22:55:41.025315 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.025320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.025329 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:41.025378 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.025380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.025396 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.025472 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:41.025557 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.026645 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.027840 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.028350 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.028404 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.028482 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.028944 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.028988 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260620 22:55:41.029148 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.025145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.2.90.129:46767)
0620 22:55:41.025375 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.025378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.025386 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:41.025424 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.025425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.025438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.025729 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:41.025787 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.026801 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.027858 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.028538 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.028597 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.028699 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.029040 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.029077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260620 22:55:41.029182 21252 tablet_service.cc:1467] Processing CreateTablet for tablet 0f63d9b880fe4b988f8cbcc678f0b46d (DEFAULT_TABLE table=test-table [id=30345a7338664a4290a7b96e6c0c2c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:41.029520 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f63d9b880fe4b988f8cbcc678f0b46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:41.029688 21383 tablet_service.cc:1467] Processing CreateTablet for tablet 0f63d9b880fe4b988f8cbcc678f0b46d (DEFAULT_TABLE table=test-table [id=30345a7338664a4290a7b96e6c0c2c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:41.029703 21121 tablet_service.cc:1467] Processing CreateTablet for tablet 0f63d9b880fe4b988f8cbcc678f0b46d (DEFAULT_TABLE table=test-table [id=30345a7338664a4290a7b96e6c0c2c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:41.029997 21383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f63d9b880fe4b988f8cbcc678f0b46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:41.029997 21121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f63d9b880fe4b988f8cbcc678f0b46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:41.031906 21463 tablet_bootstrap.cc:492] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: Bootstrap starting.
I20260620 22:55:41.032435 21463 tablet_bootstrap.cc:654] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:41.032755 21463 log.cc:824] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:41.033066 21464 tablet_bootstrap.cc:492] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Bootstrap starting.
I20260620 22:55:41.033392 21465 tablet_bootstrap.cc:492] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d: Bootstrap starting.
I20260620 22:55:41.033548 21463 tablet_bootstrap.cc:492] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: No bootstrap required, opened a new log
I20260620 22:55:41.033609 21463 ts_tablet_manager.cc:1397] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:41.033665 21464 tablet_bootstrap.cc:654] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:41.033974 21465 tablet_bootstrap.cc:654] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:41.033993 21464 log.cc:824] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:41.034283 21465 log.cc:824] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:41.034732 21464 tablet_bootstrap.cc:492] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: No bootstrap required, opened a new log
I20260620 22:55:41.034789 21464 ts_tablet_manager.cc:1397] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:41.034955 21465 tablet_bootstrap.cc:492] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d: No bootstrap required, opened a new log
I20260620 22:55:41.035036 21465 ts_tablet_manager.cc:1397] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:55:41.035995 21463 raft_consensus.cc:348] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.036175 21463 raft_consensus.cc:374] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:41.036211 21463 raft_consensus.cc:729] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c9efb2d3ea4befbac2341890a34764, State: Initialized, Role: FOLLOWER
I20260620 22:55:41.036314 21463 consensus_queue.cc:260] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.036549 21463 ts_tablet_manager.cc:1428] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:41.036664 21318 heartbeater.cc:503] Master 127.2.90.190:34165 was elected leader, sending a full tablet report...
I20260620 22:55:41.037149 21464 raft_consensus.cc:348] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.037297 21464 raft_consensus.cc:374] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:41.037340 21464 raft_consensus.cc:729] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb43b3178894aca87f013f7d1e615c8, State: Initialized, Role: FOLLOWER
I20260620 22:55:41.037263 21465 raft_consensus.cc:348] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.037417 21465 raft_consensus.cc:374] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:41.037460 21465 raft_consensus.cc:729] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2f0221028b4293827356b5320e749d, State: Initialized, Role: FOLLOWER
I20260620 22:55:41.037451 21464 consensus_queue.cc:260] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.037565 21465 consensus_queue.cc:260] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.037678 21464 ts_tablet_manager.cc:1428] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:41.037781 21449 heartbeater.cc:503] Master 127.2.90.190:34165 was elected leader, sending a full tablet report...
I20260620 22:55:41.037796 21465 ts_tablet_manager.cc:1428] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:41.037902 21187 heartbeater.cc:503] Master 127.2.90.190:34165 was elected leader, sending a full tablet report...
W20260620 22:55:41.121714 21319 tablet.cc:2369] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:41.165128 21470 raft_consensus.cc:482] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:41.165283 21470 raft_consensus.cc:504] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.166079 21470 leader_election.cc:288] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1c9efb2d3ea4befbac2341890a34764 (127.2.90.130:42797), 8f2f0221028b4293827356b5320e749d (127.2.90.129:46767)
I20260620 22:55:41.169417 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.165946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46767 (local address 127.2.90.131:54811)
0620 22:55:41.166155 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.166165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:41.166242 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:41.166398 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:41.166399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:41.166401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:41.166476 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.166478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.167436 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.167437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.168453 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.168456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.169148 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.169149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.169222 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.169226 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:41.169281 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 22:55:41.169320 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
I20260620 22:55:41.169616 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.165806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42797 (local address 127.2.90.131:37177)
0620 22:55:41.165845 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.165887 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:41.165932 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:41.166218 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:41.166219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:41.166287 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:41.166379 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.166382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.167302 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.167304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.168460 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:41.168463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.169336 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.169337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:41.169443 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.169447 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:41.169497 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:55:41.169539 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":13}
I20260620 22:55:41.169826 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.165895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:37177 (local address 127.2.90.130:42797)
0620 22:55:41.165938 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.165940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.165950 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:55:41.166084 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.166085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.166097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.166252 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:41.166466 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.167207 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.168560 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.169131 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.169200 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.169208 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.169704 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.169747 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 22:55:41.170138 21272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f63d9b880fe4b988f8cbcc678f0b46d" candidate_uuid: "ceb43b3178894aca87f013f7d1e615c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c9efb2d3ea4befbac2341890a34764" is_pre_election: true
I20260620 22:55:41.170267 21272 raft_consensus.cc:2454] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb43b3178894aca87f013f7d1e615c8 in term 0.
I20260620 22:55:41.170478 21337 leader_election.cc:302] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1c9efb2d3ea4befbac2341890a34764, ceb43b3178894aca87f013f7d1e615c8; no voters:
I20260620 22:55:41.170588 21470 raft_consensus.cc:2781] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:41.170632 21470 raft_consensus.cc:482] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:41.170651 21470 raft_consensus.cc:3037] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:41.170773 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.166080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:54811 (local address 127.2.90.129:46767)
0620 22:55:41.166187 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.166189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.166224 (+ 35us) server_negotiation.cc:407] Connection header received
0620 22:55:41.166295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.166296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.166307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.166352 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:41.166638 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.167358 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.168527 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.169037 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.169106 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.169116 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.169305 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.170697 (+ 1392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 22:55:41.171026 21141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f63d9b880fe4b988f8cbcc678f0b46d" candidate_uuid: "ceb43b3178894aca87f013f7d1e615c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2f0221028b4293827356b5320e749d" is_pre_election: true
I20260620 22:55:41.171141 21141 raft_consensus.cc:2454] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb43b3178894aca87f013f7d1e615c8 in term 0.
I20260620 22:55:41.171437 21470 raft_consensus.cc:504] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.171547 21470 leader_election.cc:288] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [CANDIDATE]: Term 1 election: Requested vote from peers c1c9efb2d3ea4befbac2341890a34764 (127.2.90.130:42797), 8f2f0221028b4293827356b5320e749d (127.2.90.129:46767)
I20260620 22:55:41.171684 21272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f63d9b880fe4b988f8cbcc678f0b46d" candidate_uuid: "ceb43b3178894aca87f013f7d1e615c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1c9efb2d3ea4befbac2341890a34764"
I20260620 22:55:41.171698 21141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f63d9b880fe4b988f8cbcc678f0b46d" candidate_uuid: "ceb43b3178894aca87f013f7d1e615c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2f0221028b4293827356b5320e749d"
I20260620 22:55:41.171756 21272 raft_consensus.cc:3037] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:41.171759 21141 raft_consensus.cc:3037] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:41.172454 21272 raft_consensus.cc:2454] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb43b3178894aca87f013f7d1e615c8 in term 1.
I20260620 22:55:41.172462 21141 raft_consensus.cc:2454] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb43b3178894aca87f013f7d1e615c8 in term 1.
I20260620 22:55:41.172613 21337 leader_election.cc:302] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f2f0221028b4293827356b5320e749d, ceb43b3178894aca87f013f7d1e615c8; no voters:
I20260620 22:55:41.172726 21470 raft_consensus.cc:2781] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:41.172874 21470 raft_consensus.cc:686] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 1 LEADER]: Becoming Leader. State: Replica: ceb43b3178894aca87f013f7d1e615c8, State: Running, Role: LEADER
I20260620 22:55:41.172961 21470 consensus_queue.cc:237] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } }
I20260620 22:55:41.173761 20999 catalog_manager.cc:5495] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb43b3178894aca87f013f7d1e615c8 (127.2.90.131). New cstate: current_term: 1 leader_uuid: "ceb43b3178894aca87f013f7d1e615c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb43b3178894aca87f013f7d1e615c8" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 38459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20260620 22:55:41.210496 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.208237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.2.90.131:38459)
0620 22:55:41.208298 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:41.208300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:41.208309 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:41.208362 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:41.208362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:41.208384 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:41.208444 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:41.208581 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.209059 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.209773 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:41.209957 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:41.210010 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:41.210170 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:41.210258 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:41.210350 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:41.210393 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 22:55:41.210400 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:55:41.212286 21272 raft_consensus.cc:1261] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 1 FOLLOWER]: Refusing update from remote peer ceb43b3178894aca87f013f7d1e615c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:41.212285 21141 raft_consensus.cc:1261] T 0f63d9b880fe4b988f8cbcc678f0b46d P 8f2f0221028b4293827356b5320e749d [term 1 FOLLOWER]: Refusing update from remote peer ceb43b3178894aca87f013f7d1e615c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:41.212664 21473 consensus_queue.cc:1035] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1c9efb2d3ea4befbac2341890a34764" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:41.212801 21470 consensus_queue.cc:1035] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f2f0221028b4293827356b5320e749d" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:41.215267 21477 mvcc.cc:205] Tried to move back new op lower bound from 7299056194402791424 to 7299056194245087232. Current Snapshot: MvccSnapshot[applied={T|T < 7299056194402791424}]
I20260620 22:55:41.216771 21478 mvcc.cc:205] Tried to move back new op lower bound from 7299056194402791424 to 7299056194245087232. Current Snapshot: MvccSnapshot[applied={T|T < 7299056194402791424 or (T in {7299056194402791424})}]
I20260620 22:55:41.217608 20998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:41.217689 20998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:41.218413 20998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90d785eed8214d29b7a57fe7a24bb1b9: Sending DeleteTablet for 3 replicas of tablet 0f63d9b880fe4b988f8cbcc678f0b46d
I20260620 22:55:41.218822 21383 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f63d9b880fe4b988f8cbcc678f0b46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:41 UTC) from {username='slave'} at 127.0.0.1:58800
I20260620 22:55:41.218954 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21059
I20260620 22:55:41.219039 21252 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f63d9b880fe4b988f8cbcc678f0b46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:41 UTC) from {username='slave'} at 127.0.0.1:52592
I20260620 22:55:41.219075 21485 tablet_replica.cc:330] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: stopping tablet replica
I20260620 22:55:41.219229 21485 raft_consensus.cc:2229] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:55:41.219345 21485 raft_consensus.cc:2258] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 22:55:41.224599 20984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:46767 (error 108)
I20260620 22:55:41.225028 21486 tablet_replica.cc:330] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764: stopping tablet replica
I20260620 22:55:41.225117 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21190
I20260620 22:55:41.225147 21486 raft_consensus.cc:2229] T 0f63d9b880fe4b988f8cbcc678f0b46d P c1c9efb2d3ea4befbac2341890a34764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:41.225282 21485 ts_tablet_manager.cc:1905] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:41.226949 21485 ts_tablet_manager.cc:1918] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:41.227067 21485 log.cc:1196] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/0f63d9b880fe4b988f8cbcc678f0b46d
I20260620 22:55:41.227109 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.226983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:46767 (local address 127.0.0.1:36394)
0620 22:55:41.227034 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.227073 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 22:55:41.227450 21485 ts_tablet_manager.cc:1939] T 0f63d9b880fe4b988f8cbcc678f0b46d P ceb43b3178894aca87f013f7d1e615c8: Deleting consensus metadata
W20260620 22:55:41.227936 20984 catalog_manager.cc:4584] TS 8f2f0221028b4293827356b5320e749d (127.2.90.129:46767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f63d9b880fe4b988f8cbcc678f0b46d: Network error: Client connection negotiation failed: client connection to 127.2.90.129:46767: connect: Connection refused (error 111)
I20260620 22:55:41.228046 20984 catalog_manager.cc:4841] TS ceb43b3178894aca87f013f7d1e615c8 (127.2.90.131:38459): tablet 0f63d9b880fe4b988f8cbcc678f0b46d (table test-table [id=30345a7338664a4290a7b96e6c0c2c14]) successfully deleted
I20260620 22:55:41.229821 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21321
I20260620 22:55:41.229894 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:41.229800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42797 (local address 127.0.0.1:52600)
0620 22:55:41.229840 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:41.229864 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:42797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:55:41.233906 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 20968
2026-06-20T22:55:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:41 dist-test-slave-4mtw krb5kdc[21490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:41 dist-test-slave-4mtw krb5kdc[21490](info): set up 2 sockets
Jun 20 22:55:41 dist-test-slave-4mtw krb5kdc[21490](info): commencing operation
krb5kdc: starting...
W20260620 22:55:42.176283 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:55:42.285775 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44209
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:42.363476 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:42.363668 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:42.363688 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:42.364771 21503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:42.364804 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:42.364816 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:42.364837 21503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:42.364853 21503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:42.366214 21503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:42.366394 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:42.366559 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:42.370954 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.370965 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.370966 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:42.371039 21503 server_base.cc:1034] running on GCE node
I20260620 22:55:42.371435 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:42.371704 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:42.372843 21503 hybrid_clock.cc:648] HybridClock initialized: now 1781996142372825 us; error 33 us; skew 500 ppm
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:42.375088 21503 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:55:42.375484 21503 webserver.cc:460] Webserver started at http://127.2.90.190:35531/ using document root <none> and password file <none>
I20260620 22:55:42.375669 21503 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:42.375712 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:42.375838 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:42.376866 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "937b4af3df4e42c2867aaffbed4efa8b"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "d84445f760cf95b125a6861055d8d93c"
server_key_iv: "84b94ee8e6401a2e23748ad77e1ea892"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.377177 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "937b4af3df4e42c2867aaffbed4efa8b"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "d84445f760cf95b125a6861055d8d93c"
server_key_iv: "84b94ee8e6401a2e23748ad77e1ea892"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.378368 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 22:55:42.378993 21518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.379140 21503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:42.379216 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "937b4af3df4e42c2867aaffbed4efa8b"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "d84445f760cf95b125a6861055d8d93c"
server_key_iv: "84b94ee8e6401a2e23748ad77e1ea892"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.379271 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:42.390506 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:42.392045 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:42.392163 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:42.395720 21503 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:44209
I20260620 22:55:42.395757 21570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:44209 every 8 connection(s)
I20260620 22:55:42.396057 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:42.396845 21571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:42.399035 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: Bootstrap starting.
I20260620 22:55:42.399529 21571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:42.399771 21571 log.cc:824] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:42.399859 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21503
I20260620 22:55:42.399956 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:42.400127 2410 external_mini_cluster.cc:1428] Setting key f26e6fdd4ae5bf9b0f8cac3a7ff2f316
I20260620 22:55:42.400446 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: No bootstrap required, opened a new log
I20260620 22:55:42.402375 21571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937b4af3df4e42c2867aaffbed4efa8b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44209 } }
I20260620 22:55:42.402544 21571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:42.402577 21571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937b4af3df4e42c2867aaffbed4efa8b, State: Initialized, Role: FOLLOWER
I20260620 22:55:42.402707 21571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937b4af3df4e42c2867aaffbed4efa8b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44209 } }
I20260620 22:55:42.402779 21571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:42.402812 21571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:42.402853 21571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:42.403496 21571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937b4af3df4e42c2867aaffbed4efa8b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44209 } }
I20260620 22:55:42.403599 21571 leader_election.cc:302] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937b4af3df4e42c2867aaffbed4efa8b; no voters:
I20260620 22:55:42.403744 21571 leader_election.cc:288] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:42.403786 21576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:42.403920 21576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [term 1 LEADER]: Becoming Leader. State: Replica: 937b4af3df4e42c2867aaffbed4efa8b, State: Running, Role: LEADER
I20260620 22:55:42.403959 21571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:42.404004 21576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937b4af3df4e42c2867aaffbed4efa8b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44209 } }
I20260620 22:55:42.404306 21576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937b4af3df4e42c2867aaffbed4efa8b. Latest consensus state: current_term: 1 leader_uuid: "937b4af3df4e42c2867aaffbed4efa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937b4af3df4e42c2867aaffbed4efa8b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44209 } } }
I20260620 22:55:42.404399 21576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:42.404320 21577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937b4af3df4e42c2867aaffbed4efa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937b4af3df4e42c2867aaffbed4efa8b" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44209 } } }
I20260620 22:55:42.404485 21577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b [sys.catalog]: This master's current role is: LEADER
W20260620 22:55:42.405778 21592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:55:42.405831 21592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:55:42.405915 21582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:42.406379 21582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:42.408257 21582 catalog_manager.cc:1347] Generated new cluster ID: 26239a7955d74bf1906d2132742bb92d
I20260620 22:55:42.408298 21582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:42.411613 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.400566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.2.90.190:44209)
0620 22:55:42.400751 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.400760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.400793 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:55:42.401636 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.401646 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.401708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.401854 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:42.402371 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.403980 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.405100 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.405703 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.405800 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.408239 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:42.408242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:42.408243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:42.408254 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:42.410102 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.410470 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.410473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.410473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.410576 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.410747 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.410747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.410748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.411041 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:42.411210 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.411279 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.411423 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260620 22:55:42.416595 21582 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:42.417277 21582 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:42.424391 21582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: Generated new TSK 0
I20260620 22:55:42.424547 21582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.90.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
Entry for principal kudu/127.2.90.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab.
I20260620 22:55:42.450762 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44209
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:42.539590 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:42.539855 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:42.539891 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:42.541597 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:42.541651 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:42.541689 21597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:42.543751 21597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.129.keytab
--principal=kudu/127.2.90.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:42.544011 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:42.544257 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:42.549079 21605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.549116 21602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.549094 21603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:42.549279 21597 server_base.cc:1034] running on GCE node
I20260620 22:55:42.549482 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:42.549718 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:42.550860 21597 hybrid_clock.cc:648] HybridClock initialized: now 1781996142550846 us; error 28 us; skew 500 ppm
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:42.555732 21597 init.cc:377] Logged in from keytab as kudu/127.2.90.129@KRBTEST.COM (short username kudu)
I20260620 22:55:42.556134 21597 webserver.cc:460] Webserver started at http://127.2.90.129:35783/ using document root <none> and password file <none>
I20260620 22:55:42.556357 21597 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:42.556407 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:42.556529 21597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:42.557592 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "6f798861fa5546c68021b6cefdcd71cd"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "a2354cbe9f5cf1eaf567c6a78e96711e"
server_key_iv: "ffab49c67c0d8e59bd121410efef07b0"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.557911 21597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "6f798861fa5546c68021b6cefdcd71cd"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "a2354cbe9f5cf1eaf567c6a78e96711e"
server_key_iv: "ffab49c67c0d8e59bd121410efef07b0"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.559089 21597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:42.559722 21612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.559870 21597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:42.559950 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "6f798861fa5546c68021b6cefdcd71cd"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "a2354cbe9f5cf1eaf567c6a78e96711e"
server_key_iv: "ffab49c67c0d8e59bd121410efef07b0"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.560015 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:42.576758 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:42.578670 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:42.578828 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:42.579049 21597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:42.579350 21597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:42.579383 21597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.579412 21597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:42.579437 21597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.585016 21597 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:38881
I20260620 22:55:42.585053 21725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:38881 every 8 connection(s)
I20260620 22:55:42.585358 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:55:42.585529 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21597
I20260620 22:55:42.585639 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:55:42.585873 2410 external_mini_cluster.cc:1428] Setting key 881f6694b576dbc0df4dec8da4bc5b34
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.129@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.130@KRBTEST.COM; defaulting to no policy
I20260620 22:55:42.594822 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.586042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:55579 (local address 127.2.90.190:44209)
0620 22:55:42.586089 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.586091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.586342 (+ 251us) server_negotiation.cc:407] Connection header received
0620 22:55:42.586857 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.586859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.586870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.586918 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:42.587955 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.588446 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.589837 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.590047 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.590105 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.592418 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:42.592419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:42.592419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:42.592427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:42.593707 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.594065 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.594066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.594066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.594113 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.594288 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.594288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.594289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.594387 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:42.594444 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.594689 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.594754 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 22:55:42.594990 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.586053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44209 (local address 127.2.90.129:55579)
0620 22:55:42.586216 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.586259 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.586764 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.586988 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.586992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.587570 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:42.587857 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.587865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.588520 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.588521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.589742 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.589746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.590126 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.590127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.590278 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.590729 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:42.590740 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:42.592331 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:42.593776 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:42.593791 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:42.593794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:42.594003 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:42.594157 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:42.594158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:42.594158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:42.594245 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:42.594443 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:42.594449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:42.594573 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.594656 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.594776 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260620 22:55:42.595471 21726 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44209
I20260620 22:55:42.595553 21726 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:42.595715 21726 heartbeater.cc:511] Master 127.2.90.190:44209 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.130@KRBTEST.COM" created.
I20260620 22:55:42.596227 21535 ts_manager.cc:194] Registered new tserver with Master: 6f798861fa5546c68021b6cefdcd71cd (127.2.90.129:38881)
I20260620 22:55:42.596865 21535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.129@KRBTEST.COM'} at 127.2.90.129:55579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
Entry for principal kudu/127.2.90.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab.
I20260620 22:55:42.609315 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44209
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:42.687110 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:42.687306 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:42.687336 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:42.688408 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:42.688439 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:42.688463 21731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:42.689836 21731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.130.keytab
--principal=kudu/127.2.90.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:42.690021 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:42.690189 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:42.694765 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:42.694811 21731 server_base.cc:1034] running on GCE node
W20260620 22:55:42.694775 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.694775 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:42.695116 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:42.695304 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:42.696431 21731 hybrid_clock.cc:648] HybridClock initialized: now 1781996142696424 us; error 27 us; skew 500 ppm
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:42.698716 21731 init.cc:377] Logged in from keytab as kudu/127.2.90.130@KRBTEST.COM (short username kudu)
I20260620 22:55:42.699123 21731 webserver.cc:460] Webserver started at http://127.2.90.130:34935/ using document root <none> and password file <none>
I20260620 22:55:42.699326 21731 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:42.699369 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:42.699496 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:42.700495 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "56b1b3c6d204401db756a817be55aa03"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "4aac53616dfd4b24aa673a0049da14c9"
server_key_iv: "e8b446977679817324cd9107bd9efe79"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.700830 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "56b1b3c6d204401db756a817be55aa03"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "4aac53616dfd4b24aa673a0049da14c9"
server_key_iv: "e8b446977679817324cd9107bd9efe79"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.702059 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:55:42.702718 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.702841 21731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:42.702916 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "56b1b3c6d204401db756a817be55aa03"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "4aac53616dfd4b24aa673a0049da14c9"
server_key_iv: "e8b446977679817324cd9107bd9efe79"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.702979 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:42.717335 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:42.718964 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:42.719092 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:42.719295 21731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:42.719583 21731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:42.719614 21731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.719641 21731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:42.719666 21731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.724975 21731 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:42763
I20260620 22:55:42.725015 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:42763 every 8 connection(s)
I20260620 22:55:42.725314 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.130@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:42.733350 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.725955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:39367 (local address 127.2.90.190:44209)
0620 22:55:42.726008 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.726010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.726199 (+ 189us) server_negotiation.cc:407] Connection header received
0620 22:55:42.726682 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.726683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.726692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.726745 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:42.727613 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.728091 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.729125 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.729306 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.729359 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.731223 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:42.731224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:42.731225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:42.731232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:42.732282 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.732602 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.732602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.732603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.732647 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.732848 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.732849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.732849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.732938 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:42.732981 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.733202 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.733270 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:55:42.733450 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.725920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44209 (local address 127.2.90.130:39367)
0620 22:55:42.726080 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.726120 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.726604 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.726806 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.726809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.727348 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:42.727544 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.727549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.728155 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.728155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.729056 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.729059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.729370 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.729370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.729468 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.729870 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:42.729878 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:42.731142 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:42.732344 (+ 1202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:42.732350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:42.732352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:42.732550 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:42.732694 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:42.732695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:42.732695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:42.732804 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:42.732980 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:42.732982 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:42.733088 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.733162 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.733278 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 22:55:42.733662 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21731
I20260620 22:55:42.733768 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:55:42.733880 21860 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44209
I20260620 22:55:42.733964 2410 external_mini_cluster.cc:1428] Setting key 6086794b47d7610e804d102a63f03ee3
I20260620 22:55:42.733978 21860 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:42.734155 21860 heartbeater.cc:511] Master 127.2.90.190:44209 requested a full tablet report, sending...
I20260620 22:55:42.734543 21534 ts_manager.cc:194] Registered new tserver with Master: 56b1b3c6d204401db756a817be55aa03 (127.2.90.130:42763)
I20260620 22:55:42.734964 21534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.130@KRBTEST.COM'} at 127.2.90.130:39367
WARNING: no policy specified for kudu/127.2.90.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
Entry for principal kudu/127.2.90.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab.
I20260620 22:55:42.755295 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44209
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:42.832343 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:42.832532 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:42.832553 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:42.833674 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:42.833712 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:42.833740 21865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:42.835114 21865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.131.keytab
--principal=kudu/127.2.90.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:42.835319 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:42.835515 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:42.840032 21871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.840096 21873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:42.840096 21870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:42.840247 21865 server_base.cc:1034] running on GCE node
I20260620 22:55:42.840445 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:42.840665 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:42.841830 21865 hybrid_clock.cc:648] HybridClock initialized: now 1781996142841817 us; error 27 us; skew 500 ppm
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:42.844000 21865 init.cc:377] Logged in from keytab as kudu/127.2.90.131@KRBTEST.COM (short username kudu)
I20260620 22:55:42.844374 21865 webserver.cc:460] Webserver started at http://127.2.90.131:43433/ using document root <none> and password file <none>
I20260620 22:55:42.844568 21865 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:42.844614 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:42.844756 21865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:42.845741 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "9ef757ecc4f84fef80043c960d4811e6"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "e4479cab665326d605f8c47706fdccce"
server_key_iv: "c4a0734a5cdd9519267c5fbf1a86e843"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.846058 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "9ef757ecc4f84fef80043c960d4811e6"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "e4479cab665326d605f8c47706fdccce"
server_key_iv: "c4a0734a5cdd9519267c5fbf1a86e843"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.847240 21865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:55:42.847847 21880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.847991 21865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:42.848069 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "9ef757ecc4f84fef80043c960d4811e6"
format_stamp: "Formatted at 2026-06-20 22:55:42 on dist-test-slave-4mtw"
server_key: "e4479cab665326d605f8c47706fdccce"
server_key_iv: "c4a0734a5cdd9519267c5fbf1a86e843"
server_key_version: "encryptionkey@0"
I20260620 22:55:42.848127 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:42.875706 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:42.877599 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:42.877739 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:42.877965 21865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:42.878268 21865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:42.878300 21865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.878329 21865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:42.878346 21865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:42.883828 21865 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:45423
I20260620 22:55:42.883890 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:45423 every 8 connection(s)
I20260620 22:55:42.884181 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:42.889613 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 21865
I20260620 22:55:42.889707 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:55:42.889866 2410 external_mini_cluster.cc:1428] Setting key ce6db6814c790cfc2fd2ee5d2cd7e6e4
Jun 20 22:55:42 dist-test-slave-4mtw krb5kdc[21490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996142, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.131@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:42.892604 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.884880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:47659 (local address 127.2.90.190:44209)
0620 22:55:42.884941 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.884943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.885164 (+ 221us) server_negotiation.cc:407] Connection header received
0620 22:55:42.885665 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.885666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.885679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.885732 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:42.886651 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.887159 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.888124 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.888322 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.888378 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.890506 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:42.890507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:42.890507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:42.890513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:42.891609 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.891935 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.891936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.891936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.891981 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.892134 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.892134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.892134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.892220 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:42.892292 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.892482 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.892542 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 22:55:42.892819 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.884877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44209 (local address 127.2.90.131:47659)
0620 22:55:42.885045 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.885081 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.885565 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.885802 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.885805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.886349 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:42.886553 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.886557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.887245 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.887246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.888038 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.888039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.888388 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.888390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.888513 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.888954 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:55:42.888962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:42.890398 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:42.891669 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:42.891675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:42.891677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:42.891880 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:42.892019 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:55:42.892019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:55:42.892020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:55:42.892102 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:55:42.892273 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:42.892275 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:42.892384 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.892460 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.892582 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 22:55:42.893195 21994 heartbeater.cc:347] Connected to a master server at 127.2.90.190:44209
I20260620 22:55:42.893306 21994 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:42.893476 21994 heartbeater.cc:511] Master 127.2.90.190:44209 requested a full tablet report, sending...
I20260620 22:55:42.893810 21534 ts_manager.cc:194] Registered new tserver with Master: 9ef757ecc4f84fef80043c960d4811e6 (127.2.90.131:45423)
I20260620 22:55:42.894160 21534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.90.131@KRBTEST.COM'} at 127.2.90.131:47659
I20260620 22:55:42.901461 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:42.908197 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.902703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.2.90.190:44209)
0620 22:55:42.902754 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.902756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.902860 (+ 104us) server_negotiation.cc:407] Connection header received
0620 22:55:42.902923 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.902924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.902933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.902977 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:42.903532 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.904002 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.904848 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.905027 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.905080 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.906231 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:42.906232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:42.906232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:42.906238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:42.907362 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.907643 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.907644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.907644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.907687 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:42.907808 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:42.907809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:42.907809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:42.907932 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:42.907998 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.908084 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.908135 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:55:42.909624 21534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:42.911115 21534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:42.916960 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.913229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:42763 (local address 127.0.0.1:55306)
0620 22:55:42.913457 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.913465 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.913523 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.913765 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.913766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.913768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:42.913839 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.913841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.914751 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.914752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.915915 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.915918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.916652 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.916763 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.916777 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.916838 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.916889 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":61,"threads_started":1}
I20260620 22:55:42.917086 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.913373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:45423 (local address 127.0.0.1:40580)
0620 22:55:42.913602 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.913612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.913665 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.913977 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.913978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.913980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:42.914018 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.914019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.915170 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.915171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.915945 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.915947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.916853 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.916923 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.916927 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.916975 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.917015 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":42,"threads_started":1}
I20260620 22:55:42.917183 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.913053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:38881 (local address 127.0.0.1:33182)
0620 22:55:42.913182 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.913225 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.913289 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.913752 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.913756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.913757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:42.913819 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.913821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.915117 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.915118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.916149 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.916151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.917010 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.917011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.917064 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.917067 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.917092 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.917123 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":50,"threads_started":1}
I20260620 22:55:42.917299 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.913461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.2.90.131:45423)
0620 22:55:42.913607 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.913612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.913655 (+ 43us) server_negotiation.cc:407] Connection header received
0620 22:55:42.913724 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.913726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.913740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.913907 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:42.914088 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.915101 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916023 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.916647 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.916827 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.917187 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.917228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260620 22:55:42.916978 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.913347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.2.90.130:42763)
0620 22:55:42.913518 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.913523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.913533 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:55:42.913607 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.913609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.913626 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.913691 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:42.913917 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.914660 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916013 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.916581 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916641 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.916722 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.916859 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.916900 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260620 22:55:42.917479 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.913189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.2.90.129:38881)
0620 22:55:42.913618 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.913622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.913630 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:42.913665 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.913667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.913680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.913749 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:42.913898 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.915015 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.916222 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.916931 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.917270 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.917336 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.917361 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.917409 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":82,"threads_started":1}
I20260620 22:55:42.917927 21928 tablet_service.cc:1467] Processing CreateTablet for tablet aa70bf5120e641658833bc47e3f7c3c0 (DEFAULT_TABLE table=test-table [id=9d7513cac89a460382b25caf113cbac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:42.917991 21794 tablet_service.cc:1467] Processing CreateTablet for tablet aa70bf5120e641658833bc47e3f7c3c0 (DEFAULT_TABLE table=test-table [id=9d7513cac89a460382b25caf113cbac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:42.918272 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa70bf5120e641658833bc47e3f7c3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:42.918272 21794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa70bf5120e641658833bc47e3f7c3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:42.921259 21660 tablet_service.cc:1467] Processing CreateTablet for tablet aa70bf5120e641658833bc47e3f7c3c0 (DEFAULT_TABLE table=test-table [id=9d7513cac89a460382b25caf113cbac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:42.921566 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa70bf5120e641658833bc47e3f7c3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:42.921566 22009 tablet_bootstrap.cc:492] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Bootstrap starting.
I20260620 22:55:42.922190 22009 tablet_bootstrap.cc:654] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:42.922731 22009 log.cc:824] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:42.923040 22008 tablet_bootstrap.cc:492] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Bootstrap starting.
I20260620 22:55:42.923408 22009 tablet_bootstrap.cc:492] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: No bootstrap required, opened a new log
I20260620 22:55:42.923467 22009 ts_tablet_manager.cc:1397] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:42.923659 22008 tablet_bootstrap.cc:654] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:42.924208 22008 log.cc:824] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:42.924785 22012 tablet_bootstrap.cc:492] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd: Bootstrap starting.
I20260620 22:55:42.925050 22009 raft_consensus.cc:348] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.925187 22009 raft_consensus.cc:374] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:42.925220 22009 raft_consensus.cc:729] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef757ecc4f84fef80043c960d4811e6, State: Initialized, Role: FOLLOWER
I20260620 22:55:42.925360 22009 consensus_queue.cc:260] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.925568 22008 tablet_bootstrap.cc:492] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: No bootstrap required, opened a new log
I20260620 22:55:42.925614 22009 ts_tablet_manager.cc:1428] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 22:55:42.925635 22008 ts_tablet_manager.cc:1397] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:55:42.925625 21994 heartbeater.cc:503] Master 127.2.90.190:44209 was elected leader, sending a full tablet report...
I20260620 22:55:42.927249 22008 raft_consensus.cc:348] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.927380 22008 raft_consensus.cc:374] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:42.927402 22008 raft_consensus.cc:729] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b1b3c6d204401db756a817be55aa03, State: Initialized, Role: FOLLOWER
I20260620 22:55:42.927518 22008 consensus_queue.cc:260] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.927762 22008 ts_tablet_manager.cc:1428] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 22:55:42.927770 21860 heartbeater.cc:503] Master 127.2.90.190:44209 was elected leader, sending a full tablet report...
I20260620 22:55:42.927964 22012 tablet_bootstrap.cc:654] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:42.928504 22012 log.cc:824] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:42.929167 22012 tablet_bootstrap.cc:492] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd: No bootstrap required, opened a new log
I20260620 22:55:42.929224 22012 ts_tablet_manager.cc:1397] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 22:55:42.930562 22012 raft_consensus.cc:348] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.930687 22012 raft_consensus.cc:374] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:42.930717 22012 raft_consensus.cc:729] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f798861fa5546c68021b6cefdcd71cd, State: Initialized, Role: FOLLOWER
I20260620 22:55:42.930835 22012 consensus_queue.cc:260] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.931046 22012 ts_tablet_manager.cc:1428] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 22:55:42.931077 21726 heartbeater.cc:503] Master 127.2.90.190:44209 was elected leader, sending a full tablet report...
W20260620 22:55:42.975829 21861 tablet.cc:2369] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:42.984855 22014 raft_consensus.cc:482] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:42.985006 22014 raft_consensus.cc:504] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.985396 22014 leader_election.cc:288] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f798861fa5546c68021b6cefdcd71cd (127.2.90.129:38881), 9ef757ecc4f84fef80043c960d4811e6 (127.2.90.131:45423)
I20260620 22:55:42.989112 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.985640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:34443 (local address 127.2.90.129:38881)
0620 22:55:42.985688 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.985690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.985697 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:55:42.985724 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.985725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.985738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.985788 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:42.985951 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.986707 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.988151 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.988769 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.988844 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.988853 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.989004 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.989041 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:55:42.989112 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.985732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:38381 (local address 127.2.90.131:45423)
0620 22:55:42.985787 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:42.985789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:42.985931 (+ 142us) server_negotiation.cc:407] Connection header received
0620 22:55:42.985977 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:42.985978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:42.985987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:42.986040 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:42.986132 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.986791 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.988148 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.988775 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.988847 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.988854 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:42.989004 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 22:55:42.989041 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:55:42.989131 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.985556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:38881 (local address 127.2.90.130:34443)
0620 22:55:42.985613 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.985627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.985674 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.985803 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.985805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.985807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:42.985884 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.985886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.986777 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.986778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.988064 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.988066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.988846 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.988847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.988935 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.988937 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.988998 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.989034 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":54}
I20260620 22:55:42.989137 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:42.985676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:45423 (local address 127.2.90.130:38381)
0620 22:55:42.985860 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:42.985870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:42.985928 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:42.986043 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:42.986043 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:42.986045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:42.986092 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.986093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.986853 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.986853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.988064 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:42.988066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:42.988846 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:42.988847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:42.988932 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:42.988937 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:55:42.988993 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 22:55:42.989034 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":39,"thread_start_us":110,"threads_started":1}
I20260620 22:55:42.989568 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa70bf5120e641658833bc47e3f7c3c0" candidate_uuid: "56b1b3c6d204401db756a817be55aa03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f798861fa5546c68021b6cefdcd71cd" is_pre_election: true
I20260620 22:55:42.989573 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa70bf5120e641658833bc47e3f7c3c0" candidate_uuid: "56b1b3c6d204401db756a817be55aa03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef757ecc4f84fef80043c960d4811e6" is_pre_election: true
I20260620 22:55:42.989739 21680 raft_consensus.cc:2454] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b1b3c6d204401db756a817be55aa03 in term 0.
I20260620 22:55:42.989739 21948 raft_consensus.cc:2454] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56b1b3c6d204401db756a817be55aa03 in term 0.
I20260620 22:55:42.989966 21749 leader_election.cc:302] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b1b3c6d204401db756a817be55aa03, 6f798861fa5546c68021b6cefdcd71cd; no voters:
I20260620 22:55:42.990111 22014 raft_consensus.cc:2781] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:42.990181 22014 raft_consensus.cc:482] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:42.990198 22014 raft_consensus.cc:3037] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:42.990868 22014 raft_consensus.cc:504] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.990988 22014 leader_election.cc:288] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [CANDIDATE]: Term 1 election: Requested vote from peers 6f798861fa5546c68021b6cefdcd71cd (127.2.90.129:38881), 9ef757ecc4f84fef80043c960d4811e6 (127.2.90.131:45423)
I20260620 22:55:42.991154 21680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa70bf5120e641658833bc47e3f7c3c0" candidate_uuid: "56b1b3c6d204401db756a817be55aa03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f798861fa5546c68021b6cefdcd71cd"
I20260620 22:55:42.991169 21948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa70bf5120e641658833bc47e3f7c3c0" candidate_uuid: "56b1b3c6d204401db756a817be55aa03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef757ecc4f84fef80043c960d4811e6"
I20260620 22:55:42.991222 21680 raft_consensus.cc:3037] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:42.991245 21948 raft_consensus.cc:3037] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:42.991888 21680 raft_consensus.cc:2454] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b1b3c6d204401db756a817be55aa03 in term 1.
I20260620 22:55:42.991925 21948 raft_consensus.cc:2454] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56b1b3c6d204401db756a817be55aa03 in term 1.
I20260620 22:55:42.992036 21749 leader_election.cc:302] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b1b3c6d204401db756a817be55aa03, 6f798861fa5546c68021b6cefdcd71cd; no voters:
I20260620 22:55:42.992138 22014 raft_consensus.cc:2781] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:42.992290 22014 raft_consensus.cc:686] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 1 LEADER]: Becoming Leader. State: Replica: 56b1b3c6d204401db756a817be55aa03, State: Running, Role: LEADER
I20260620 22:55:42.992370 22014 consensus_queue.cc:237] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } }
I20260620 22:55:42.993160 21534 catalog_manager.cc:5495] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56b1b3c6d204401db756a817be55aa03 (127.2.90.130). New cstate: current_term: 1 leader_uuid: "56b1b3c6d204401db756a817be55aa03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56b1b3c6d204401db756a817be55aa03" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 42763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20260620 22:55:43.007513 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.005242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.2.90.130:42763)
0620 22:55:43.005291 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.005292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.005300 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:43.005360 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.005360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.005379 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.005441 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:43.005598 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.006070 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.006796 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.007004 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.007056 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.007200 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:43.007307 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:43.007376 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.007402 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.007421 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:55:43.009162 21948 raft_consensus.cc:1261] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 1 FOLLOWER]: Refusing update from remote peer 56b1b3c6d204401db756a817be55aa03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:43.009186 21680 raft_consensus.cc:1261] T aa70bf5120e641658833bc47e3f7c3c0 P 6f798861fa5546c68021b6cefdcd71cd [term 1 FOLLOWER]: Refusing update from remote peer 56b1b3c6d204401db756a817be55aa03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:43.009486 22018 consensus_queue.cc:1035] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef757ecc4f84fef80043c960d4811e6" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:43.009616 22014 consensus_queue.cc:1035] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f798861fa5546c68021b6cefdcd71cd" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:43.011716 22021 mvcc.cc:205] Tried to move back new op lower bound from 7299056201762971648 to 7299056201697284096. Current Snapshot: MvccSnapshot[applied={T|T < 7299056201762971648}]
I20260620 22:55:43.011907 22022 mvcc.cc:205] Tried to move back new op lower bound from 7299056201762971648 to 7299056201697284096. Current Snapshot: MvccSnapshot[applied={T|T < 7299056201762971648}]
I20260620 22:55:43.014102 21534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:43.014186 21534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:43.014820 21534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 937b4af3df4e42c2867aaffbed4efa8b: Sending DeleteTablet for 3 replicas of tablet aa70bf5120e641658833bc47e3f7c3c0
I20260620 22:55:43.015095 21660 tablet_service.cc:1514] Processing DeleteTablet for tablet aa70bf5120e641658833bc47e3f7c3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:43 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:33182
I20260620 22:55:43.015168 21794 tablet_service.cc:1514] Processing DeleteTablet for tablet aa70bf5120e641658833bc47e3f7c3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:43 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:55306
I20260620 22:55:43.015350 22031 tablet_replica.cc:330] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: stopping tablet replica
I20260620 22:55:43.015352 21928 tablet_service.cc:1514] Processing DeleteTablet for tablet aa70bf5120e641658833bc47e3f7c3c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:43 UTC) from {username='kudu', principal='kudu/127.2.90.190@KRBTEST.COM'} at 127.0.0.1:40580
I20260620 22:55:43.015439 22031 raft_consensus.cc:2229] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:55:43.015515 22031 raft_consensus.cc:2258] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:43.015611 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21597
I20260620 22:55:43.016384 22032 tablet_replica.cc:330] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: stopping tablet replica
I20260620 22:55:43.016480 22032 raft_consensus.cc:2229] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:43.016568 22032 raft_consensus.cc:2258] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:43.017005 22032 ts_tablet_manager.cc:1905] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:43.017624 22031 ts_tablet_manager.cc:1905] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:43.018445 22032 ts_tablet_manager.cc:1918] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:43.018489 22032 log.cc:1196] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/aa70bf5120e641658833bc47e3f7c3c0
I20260620 22:55:43.018663 22032 ts_tablet_manager.cc:1939] T aa70bf5120e641658833bc47e3f7c3c0 P 9ef757ecc4f84fef80043c960d4811e6: Deleting consensus metadata
I20260620 22:55:43.018975 21521 catalog_manager.cc:4841] TS 9ef757ecc4f84fef80043c960d4811e6 (127.2.90.131:45423): tablet aa70bf5120e641658833bc47e3f7c3c0 (table test-table [id=9d7513cac89a460382b25caf113cbac9]) successfully deleted
I20260620 22:55:43.019083 22031 ts_tablet_manager.cc:1918] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:43.019140 22031 log.cc:1196] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781995912840861-2410-0/minicluster-data/ts-1/wal/wals/aa70bf5120e641658833bc47e3f7c3c0
I20260620 22:55:43.019372 22031 ts_tablet_manager.cc:1939] T aa70bf5120e641658833bc47e3f7c3c0 P 56b1b3c6d204401db756a817be55aa03: Deleting consensus metadata
I20260620 22:55:43.019773 21521 catalog_manager.cc:4841] TS 56b1b3c6d204401db756a817be55aa03 (127.2.90.130:42763): tablet aa70bf5120e641658833bc47e3f7c3c0 (table test-table [id=9d7513cac89a460382b25caf113cbac9]) successfully deleted
W20260620 22:55:43.020566 21521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:38881 (error 108)
I20260620 22:55:43.020977 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.020871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:38881 (local address 127.0.0.1:33194)
0620 22:55:43.020916 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.020950 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:55:43.021072 21521 catalog_manager.cc:4584] TS 6f798861fa5546c68021b6cefdcd71cd (127.2.90.129:38881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa70bf5120e641658833bc47e3f7c3c0: Network error: Client connection negotiation failed: client connection to 127.2.90.129:38881: connect: Connection refused (error 111)
I20260620 22:55:43.021198 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21731
I20260620 22:55:43.025528 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21865
I20260620 22:55:43.029698 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 21503
2026-06-20T22:55:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1796 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T22:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:43Z Disabled control of system clock
I20260620 22:55:43.054706 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:38129
--webserver_interface=127.2.90.190
--webserver_port=0
--builtin_ntp_servers=127.2.90.148:46877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:43.132102 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:43.132323 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:43.132342 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:43.133457 22037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:43.133492 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:43.133503 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:43.133514 22037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:43.133527 22037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:43.134788 22037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:38129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:43.134974 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:43.135205 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:43.139590 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:43.139590 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:43.139685 22037 server_base.cc:1034] running on GCE node
W20260620 22:55:43.139665 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:43.140092 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:43.140386 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:43.141520 22037 hybrid_clock.cc:648] HybridClock initialized: now 1781996143141505 us; error 27 us; skew 500 ppm
I20260620 22:55:43.142611 22037 webserver.cc:460] Webserver started at http://127.2.90.190:41505/ using document root <none> and password file <none>
I20260620 22:55:43.142838 22037 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:43.142884 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:43.143009 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:43.144001 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "ffa5fe264b324912a0fe75112e42e701"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "baf05200c18528ba4af0a57bc7b863ee"
server_key_iv: "cc7dde2788ad5e072716a5185c579815"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.144315 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "ffa5fe264b324912a0fe75112e42e701"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "baf05200c18528ba4af0a57bc7b863ee"
server_key_iv: "cc7dde2788ad5e072716a5185c579815"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.145503 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:55:43.146123 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.146277 22037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:43.146369 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "ffa5fe264b324912a0fe75112e42e701"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "baf05200c18528ba4af0a57bc7b863ee"
server_key_iv: "cc7dde2788ad5e072716a5185c579815"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.146432 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:43.167382 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:43.167763 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:43.167924 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:43.172034 22037 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:38129
I20260620 22:55:43.172096 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:38129 every 8 connection(s)
I20260620 22:55:43.172372 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:43.173316 22104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:43.175621 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701: Bootstrap starting.
I20260620 22:55:43.176184 22104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:43.176455 22104 log.cc:824] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:43.177112 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701: No bootstrap required, opened a new log
I20260620 22:55:43.178222 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22037
I20260620 22:55:43.178314 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:43.178473 2410 external_mini_cluster.cc:1428] Setting key 90da782aebaf029060da8f51ed9249c4
I20260620 22:55:43.178794 22104 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa5fe264b324912a0fe75112e42e701" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 38129 } }
I20260620 22:55:43.178992 22104 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:43.179039 22104 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa5fe264b324912a0fe75112e42e701, State: Initialized, Role: FOLLOWER
I20260620 22:55:43.179173 22104 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa5fe264b324912a0fe75112e42e701" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 38129 } }
I20260620 22:55:43.179258 22104 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:43.179287 22104 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:43.179325 22104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:43.180017 22104 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa5fe264b324912a0fe75112e42e701" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 38129 } }
I20260620 22:55:43.180128 22104 leader_election.cc:302] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa5fe264b324912a0fe75112e42e701; no voters:
I20260620 22:55:43.180294 22104 leader_election.cc:288] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:43.180346 22109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:43.180495 22109 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [term 1 LEADER]: Becoming Leader. State: Replica: ffa5fe264b324912a0fe75112e42e701, State: Running, Role: LEADER
I20260620 22:55:43.180546 22104 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:43.180588 22109 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa5fe264b324912a0fe75112e42e701" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 38129 } }
I20260620 22:55:43.180951 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa5fe264b324912a0fe75112e42e701. Latest consensus state: current_term: 1 leader_uuid: "ffa5fe264b324912a0fe75112e42e701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa5fe264b324912a0fe75112e42e701" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 38129 } } }
I20260620 22:55:43.181018 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:43.181020 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa5fe264b324912a0fe75112e42e701" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa5fe264b324912a0fe75112e42e701" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 38129 } } }
I20260620 22:55:43.181177 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:43.181912 22114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:43.182478 22114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:43.184151 22114 catalog_manager.cc:1347] Generated new cluster ID: 8e8ce4c887254edb8ad52bb9092316bf
I20260620 22:55:43.184202 22114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:43.184386 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.179022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.2.90.190:38129)
0620 22:55:43.179258 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.179268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.179308 (+ 40us) server_negotiation.cc:407] Connection header received
0620 22:55:43.179426 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.179434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.179514 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.179663 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:43.179840 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.181644 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.182613 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.183026 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.183135 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.183760 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:43.183764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:43.183764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:43.183930 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:43.183963 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:43.183977 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:43.184036 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.184122 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.184202 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260620 22:55:43.223460 22114 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:43.224171 22114 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:43.231360 22114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701: Generated new TSK 0
I20260620 22:55:43.231549 22114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:43.241401 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.129:0
--local_ip_for_outbound_sockets=127.2.90.129
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:38129
--builtin_ntp_servers=127.2.90.148:46877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:43.319715 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:43.319901 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:43.319921 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:43.321062 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:43.321099 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:43.321122 22128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.129
I20260620 22:55:43.322427 22128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.90.129
--webserver_port=0
--tserver_master_addrs=127.2.90.190:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.129
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:43.322638 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:43.322896 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:43.327543 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:43.327612 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:43.327538 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:43.327664 22128 server_base.cc:1034] running on GCE node
I20260620 22:55:43.327921 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:43.328130 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:43.329309 22128 hybrid_clock.cc:648] HybridClock initialized: now 1781996143329294 us; error 26 us; skew 500 ppm
I20260620 22:55:43.330466 22128 webserver.cc:460] Webserver started at http://127.2.90.129:33137/ using document root <none> and password file <none>
I20260620 22:55:43.330680 22128 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:43.330727 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:43.330848 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:43.331817 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data/instance:
uuid: "f3f51c131a6e477a899ff8992574f8af"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "2720acb6a7e4ed289600d8b40da93f61"
server_key_iv: "8f9827d45899063bcc6e302690e0a5c7"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.332123 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f51c131a6e477a899ff8992574f8af"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "2720acb6a7e4ed289600d8b40da93f61"
server_key_iv: "8f9827d45899063bcc6e302690e0a5c7"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.333349 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:43.333945 22142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.334095 22128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:43.334170 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal
uuid: "f3f51c131a6e477a899ff8992574f8af"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "2720acb6a7e4ed289600d8b40da93f61"
server_key_iv: "8f9827d45899063bcc6e302690e0a5c7"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.334229 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:43.354089 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:43.354384 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:43.354508 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:43.354708 22128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:43.355020 22128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:43.355053 22128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.355082 22128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:43.355098 22128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.360208 22128 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.129:38745
I20260620 22:55:43.360275 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.129:38745 every 8 connection(s)
I20260620 22:55:43.360553 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/data/info.pb
I20260620 22:55:43.365201 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.361237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.129:43817 (local address 127.2.90.190:38129)
0620 22:55:43.361298 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.361300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.361517 (+ 217us) server_negotiation.cc:407] Connection header received
0620 22:55:43.361971 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.361972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.361983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.362035 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:43.362646 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.363131 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.364037 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.364207 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.364248 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.365012 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:43.365013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:43.365013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:43.365016 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:43.365027 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:43.365030 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:43.365062 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.365120 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.365144 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 22:55:43.365377 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.361206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:38129 (local address 127.2.90.129:43817)
0620 22:55:43.361400 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.361440 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.361901 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.362100 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.362104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.362112 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:43.362585 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.362589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.363195 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.363195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.363980 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.363982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.364259 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.364260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.364367 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.364879 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:43.364879 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:43.364908 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:43.364931 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:43.365070 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:43.365073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:43.365075 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.365118 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.365190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260620 22:55:43.365815 22256 heartbeater.cc:347] Connected to a master server at 127.2.90.190:38129
I20260620 22:55:43.365899 22256 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:43.366088 22256 heartbeater.cc:511] Master 127.2.90.190:38129 requested a full tablet report, sending...
I20260620 22:55:43.366108 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22128
I20260620 22:55:43.366206 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-0/wal/instance
I20260620 22:55:43.366382 2410 external_mini_cluster.cc:1428] Setting key 0d0a869c8dcec702bc2af29e2783154b
I20260620 22:55:43.366595 22068 ts_manager.cc:194] Registered new tserver with Master: f3f51c131a6e477a899ff8992574f8af (127.2.90.129:38745)
I20260620 22:55:43.367337 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.130:0
--local_ip_for_outbound_sockets=127.2.90.130
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:38129
--builtin_ntp_servers=127.2.90.148:46877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 22:55:43.367357 22068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.129:43817
W20260620 22:55:43.444541 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:43.444759 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:43.444780 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:43.445911 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:43.445950 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:43.445977 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.130
I20260620 22:55:43.447322 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.90.130
--webserver_port=0
--tserver_master_addrs=127.2.90.190:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.130
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:43.447546 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:43.447811 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:55:43.452773 22259 server_base.cc:1034] running on GCE node
W20260620 22:55:43.452739 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:43.452749 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:43.452745 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:43.453096 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:43.453284 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:43.454411 22259 hybrid_clock.cc:648] HybridClock initialized: now 1781996143454398 us; error 32 us; skew 500 ppm
I20260620 22:55:43.455523 22259 webserver.cc:460] Webserver started at http://127.2.90.130:43215/ using document root <none> and password file <none>
I20260620 22:55:43.455737 22259 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:43.455782 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:43.455901 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:43.456980 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data/instance:
uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "56bbbecac9abb7a4c8167b0aab3372e3"
server_key_iv: "0d972ed93a82cb15ccf3d6a7ebe70d63"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.457288 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance:
uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "56bbbecac9abb7a4c8167b0aab3372e3"
server_key_iv: "0d972ed93a82cb15ccf3d6a7ebe70d63"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.458439 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:43.459066 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.459187 22259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:43.459261 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal
uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "56bbbecac9abb7a4c8167b0aab3372e3"
server_key_iv: "0d972ed93a82cb15ccf3d6a7ebe70d63"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.459323 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:43.476857 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:43.477151 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:43.477273 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:43.477475 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:43.477797 22259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:43.477829 22259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.477859 22259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:43.477882 22259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.483399 22259 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.130:35993
I20260620 22:55:43.483453 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.130:35993 every 8 connection(s)
I20260620 22:55:43.483749 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/data/info.pb
I20260620 22:55:43.488257 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.484358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.130:41575 (local address 127.2.90.190:38129)
0620 22:55:43.484442 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.484445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.484627 (+ 182us) server_negotiation.cc:407] Connection header received
0620 22:55:43.485134 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.485136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.485148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.485201 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:43.485810 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.486285 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.487179 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.487356 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.487398 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.488052 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:43.488054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:43.488054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:43.488058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:43.488071 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:43.488075 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:43.488111 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.488175 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.488199 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 22:55:43.488416 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.484336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:38129 (local address 127.2.90.130:41575)
0620 22:55:43.484514 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.484547 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.485052 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.485254 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.485257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.485265 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:43.485749 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.485753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.486353 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.486354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.487120 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.487122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.487412 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.487413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.487512 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.487937 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:43.487938 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:43.487963 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:43.487984 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:43.488113 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:43.488115 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:43.488117 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.488168 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.488239 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260620 22:55:43.488873 22387 heartbeater.cc:347] Connected to a master server at 127.2.90.190:38129
I20260620 22:55:43.488956 22387 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:43.489121 22387 heartbeater.cc:511] Master 127.2.90.190:38129 requested a full tablet report, sending...
I20260620 22:55:43.489461 22068 ts_manager.cc:194] Registered new tserver with Master: d03b640fd9cc4ac6b45078ec7aafc2fe (127.2.90.130:35993)
I20260620 22:55:43.489816 22068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.130:41575
I20260620 22:55:43.491712 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22259
I20260620 22:55:43.491806 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-1/wal/instance
I20260620 22:55:43.491961 2410 external_mini_cluster.cc:1428] Setting key 7c9194e0e3819d8ee23c5120811958c9
I20260620 22:55:43.492805 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.90.131:0
--local_ip_for_outbound_sockets=127.2.90.131
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:38129
--builtin_ntp_servers=127.2.90.148:46877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 22:55:43.569694 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:43.569887 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:43.569917 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:43.571028 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:43.571065 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:43.571097 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.90.131
I20260620 22:55:43.572403 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:46877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.90.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.90.131
--webserver_port=0
--tserver_master_addrs=127.2.90.190:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.90.131
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:43.572629 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:43.572909 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:43.577394 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:43.577438 22390 server_base.cc:1034] running on GCE node
W20260620 22:55:43.577400 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:43.577409 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:43.577773 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:43.578003 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:43.579145 22390 hybrid_clock.cc:648] HybridClock initialized: now 1781996143579132 us; error 26 us; skew 500 ppm
I20260620 22:55:43.580300 22390 webserver.cc:460] Webserver started at http://127.2.90.131:37161/ using document root <none> and password file <none>
I20260620 22:55:43.580531 22390 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:43.580578 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:43.580726 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:43.581786 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data/instance:
uuid: "4dafc2cab4fc4235805a1126681d398b"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "863f4df252eba48d190639aa3693bbd5"
server_key_iv: "3e1ae6bba200c20032d076cd1f924235"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.582098 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance:
uuid: "4dafc2cab4fc4235805a1126681d398b"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "863f4df252eba48d190639aa3693bbd5"
server_key_iv: "3e1ae6bba200c20032d076cd1f924235"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.583216 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:43.583820 22404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.583953 22390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:43.584029 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal
uuid: "4dafc2cab4fc4235805a1126681d398b"
format_stamp: "Formatted at 2026-06-20 22:55:43 on dist-test-slave-4mtw"
server_key: "863f4df252eba48d190639aa3693bbd5"
server_key_iv: "3e1ae6bba200c20032d076cd1f924235"
server_key_version: "encryptionkey@0"
I20260620 22:55:43.584092 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:43.594239 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:43.594468 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:43.594578 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:43.594764 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:55:43.595038 22390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:55:43.595070 22390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.595096 22390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:55:43.595120 22390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:43.600164 22390 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.131:44793
I20260620 22:55:43.600222 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.131:44793 every 8 connection(s)
I20260620 22:55:43.600486 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/data/info.pb
I20260620 22:55:43.605003 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.601088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:36997 (local address 127.2.90.190:38129)
0620 22:55:43.601141 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.601143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.601336 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:55:43.601774 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.601775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.601784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.601839 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:43.602501 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.602988 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.603853 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.604026 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.604066 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.604815 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:43.604816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:43.604816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:43.604820 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:43.604830 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:43.604834 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:43.604869 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.604925 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.604945 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:55:43.605178 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.601057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:38129 (local address 127.2.90.131:36997)
0620 22:55:43.601223 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.601260 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.601699 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.601903 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.601906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.601914 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:55:43.602412 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.602416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.603048 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.603048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.603792 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.603794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.604075 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.604076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.604175 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.604656 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 22:55:43.604656 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:55:43.604719 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 22:55:43.604746 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:55:43.604871 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:55:43.604873 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:55:43.604876 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.604918 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.604997 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260620 22:55:43.605561 22518 heartbeater.cc:347] Connected to a master server at 127.2.90.190:38129
I20260620 22:55:43.605639 22518 heartbeater.cc:464] Registering TS with master...
I20260620 22:55:43.605767 22518 heartbeater.cc:511] Master 127.2.90.190:38129 requested a full tablet report, sending...
I20260620 22:55:43.606055 22068 ts_manager.cc:194] Registered new tserver with Master: 4dafc2cab4fc4235805a1126681d398b (127.2.90.131:44793)
I20260620 22:55:43.606367 22068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.90.131:36997
I20260620 22:55:43.606992 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22390
I20260620 22:55:43.607077 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal/instance
I20260620 22:55:43.607216 2410 external_mini_cluster.cc:1428] Setting key ac1567d878c18ea7332c13801cb991ff
I20260620 22:55:43.607975 2410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:55:43.611526 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.609160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.2.90.190:38129)
0620 22:55:43.609201 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.609203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.609287 (+ 84us) server_negotiation.cc:407] Connection header received
0620 22:55:43.609324 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.609324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.609334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.609375 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:43.609497 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.609928 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.610593 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.610770 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.610820 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.611356 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:43.611357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:43.611357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:43.611361 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:43.611372 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:43.611376 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:43.611413 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.611450 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.611468 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:55:43.612931 22068 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:55:43.614403 22068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:55:43.620060 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.616268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35993 (local address 127.0.0.1:43094)
0620 22:55:43.616744 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.616754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.616817 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.616974 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.616975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.616976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:43.617005 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.617005 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.618185 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.618186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.618924 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.618926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.619794 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.619866 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.619880 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.619942 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.619990 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260620 22:55:43.620082 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.616468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.2.90.130:35993)
0620 22:55:43.616638 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.616641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.616828 (+ 187us) server_negotiation.cc:407] Connection header received
0620 22:55:43.616880 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.616882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.616899 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.616970 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:43.617061 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.618119 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619003 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.619712 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619782 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.619868 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.619955 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.619989 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260620 22:55:43.620311 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.616237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.131:44793 (local address 127.0.0.1:51244)
0620 22:55:43.616394 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.616429 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.616491 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.616724 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.616735 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.616737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:43.616789 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.616790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.618007 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.618008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.619213 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.619216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.619972 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.620022 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.620025 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.620067 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.620096 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":29,"thread_start_us":48,"threads_started":1}
I20260620 22:55:43.620432 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.616239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:38745 (local address 127.0.0.1:58628)
0620 22:55:43.616582 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.616590 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.616624 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.616900 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.616901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.616903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:43.616942 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.616943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.617902 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.617903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.619272 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.619274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.620201 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.620202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.620266 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.620270 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.620324 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.620364 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
I20260620 22:55:43.620515 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.616306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.2.90.131:44793)
0620 22:55:43.616508 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.616513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.616524 (+ 11us) server_negotiation.cc:407] Connection header received
0620 22:55:43.616559 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.616561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.616576 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.616645 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:43.616861 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.617931 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619301 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.619799 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619860 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.619941 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.620404 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.620442 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":64,"threads_started":1}
I20260620 22:55:43.620779 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.616454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.2.90.129:38745)
0620 22:55:43.616624 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.616628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.616662 (+ 34us) server_negotiation.cc:407] Connection header received
0620 22:55:43.616736 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.616738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.616762 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.616834 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:43.617012 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.617831 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.619329 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.620025 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.620085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.620165 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.620636 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.620700 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260620 22:55:43.620958 22321 tablet_service.cc:1467] Processing CreateTablet for tablet 27e21bb772ff484d85af80b2bbf731d7 (DEFAULT_TABLE table=test-table [id=04bd64b300d4446f9d7dc05ff14a6a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:43.621302 22321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e21bb772ff484d85af80b2bbf731d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:43.621447 22190 tablet_service.cc:1467] Processing CreateTablet for tablet 27e21bb772ff484d85af80b2bbf731d7 (DEFAULT_TABLE table=test-table [id=04bd64b300d4446f9d7dc05ff14a6a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:43.621762 22190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e21bb772ff484d85af80b2bbf731d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:43.624490 22532 tablet_bootstrap.cc:492] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Bootstrap starting.
I20260620 22:55:43.624838 22533 tablet_bootstrap.cc:492] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af: Bootstrap starting.
I20260620 22:55:43.624869 22452 tablet_service.cc:1467] Processing CreateTablet for tablet 27e21bb772ff484d85af80b2bbf731d7 (DEFAULT_TABLE table=test-table [id=04bd64b300d4446f9d7dc05ff14a6a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:55:43.625173 22452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27e21bb772ff484d85af80b2bbf731d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:43.625329 22532 tablet_bootstrap.cc:654] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:43.625427 22533 tablet_bootstrap.cc:654] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:43.625634 22532 log.cc:824] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:43.625710 22533 log.cc:824] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:43.626304 22532 tablet_bootstrap.cc:492] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: No bootstrap required, opened a new log
I20260620 22:55:43.626358 22532 ts_tablet_manager.cc:1397] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:43.627686 22536 tablet_bootstrap.cc:492] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Bootstrap starting.
I20260620 22:55:43.628106 22533 tablet_bootstrap.cc:492] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af: No bootstrap required, opened a new log
I20260620 22:55:43.628165 22533 ts_tablet_manager.cc:1397] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 22:55:43.628264 22536 tablet_bootstrap.cc:654] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:43.628569 22536 log.cc:824] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:43.628577 22532 raft_consensus.cc:348] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.628752 22532 raft_consensus.cc:374] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:43.628794 22532 raft_consensus.cc:729] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03b640fd9cc4ac6b45078ec7aafc2fe, State: Initialized, Role: FOLLOWER
I20260620 22:55:43.628899 22532 consensus_queue.cc:260] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.629105 22532 ts_tablet_manager.cc:1428] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:43.629163 22387 heartbeater.cc:503] Master 127.2.90.190:38129 was elected leader, sending a full tablet report...
I20260620 22:55:43.629639 22536 tablet_bootstrap.cc:492] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: No bootstrap required, opened a new log
I20260620 22:55:43.629704 22536 ts_tablet_manager.cc:1397] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 22:55:43.630473 22533 raft_consensus.cc:348] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.630591 22533 raft_consensus.cc:374] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:43.630620 22533 raft_consensus.cc:729] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f51c131a6e477a899ff8992574f8af, State: Initialized, Role: FOLLOWER
I20260620 22:55:43.630728 22533 consensus_queue.cc:260] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.630956 22533 ts_tablet_manager.cc:1428] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:43.630990 22256 heartbeater.cc:503] Master 127.2.90.190:38129 was elected leader, sending a full tablet report...
I20260620 22:55:43.631848 22536 raft_consensus.cc:348] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.631995 22536 raft_consensus.cc:374] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:43.632037 22536 raft_consensus.cc:729] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dafc2cab4fc4235805a1126681d398b, State: Initialized, Role: FOLLOWER
I20260620 22:55:43.632130 22536 consensus_queue.cc:260] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.632313 22536 ts_tablet_manager.cc:1428] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 22:55:43.632370 22518 heartbeater.cc:503] Master 127.2.90.190:38129 was elected leader, sending a full tablet report...
I20260620 22:55:43.698603 22540 raft_consensus.cc:482] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 22:55:43.698729 22540 raft_consensus.cc:504] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.699088 22540 leader_election.cc:288] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f51c131a6e477a899ff8992574f8af (127.2.90.129:38745), d03b640fd9cc4ac6b45078ec7aafc2fe (127.2.90.130:35993)
I20260620 22:55:43.702792 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.699298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:44001 (local address 127.2.90.129:38745)
0620 22:55:43.699347 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.699349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.699381 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:55:43.699409 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.699410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.699422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.699465 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:43.699635 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.700396 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.701885 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.702450 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.702510 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.702515 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.702682 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.702720 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:55:43.702792 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.699315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.90.131:34495 (local address 127.2.90.130:35993)
0620 22:55:43.699356 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.699358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.699501 (+ 143us) server_negotiation.cc:407] Connection header received
0620 22:55:43.699534 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.699535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.699542 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.699580 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 22:55:43.699702 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.700408 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.701897 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.702436 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.702505 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.702513 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.702681 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.702720 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:55:43.702801 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.699244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:38745 (local address 127.2.90.131:44001)
0620 22:55:43.699286 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.699298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.699339 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.699497 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.699497 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.699499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:43.699567 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.699569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.700484 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.700485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.701807 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.701810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.702516 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.702517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.702617 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.702622 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.702672 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.702714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":70}
I20260620 22:55:43.702813 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.699248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35993 (local address 127.2.90.131:34495)
0620 22:55:43.699437 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.699444 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 22:55:43.699491 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:55:43.699593 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:55:43.699594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:55:43.699595 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 22:55:43.699648 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.699649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.700483 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.700484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.701807 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:55:43.701810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.702517 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.702518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:55:43.702617 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.702623 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 22:55:43.702672 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 22:55:43.702713 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":98,"thread_start_us":99,"threads_started":1}
I20260620 22:55:43.703155 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e21bb772ff484d85af80b2bbf731d7" candidate_uuid: "4dafc2cab4fc4235805a1126681d398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f51c131a6e477a899ff8992574f8af" is_pre_election: true
I20260620 22:55:43.703222 22341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e21bb772ff484d85af80b2bbf731d7" candidate_uuid: "4dafc2cab4fc4235805a1126681d398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" is_pre_election: true
I20260620 22:55:43.703296 22210 raft_consensus.cc:2454] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dafc2cab4fc4235805a1126681d398b in term 0.
I20260620 22:55:43.703382 22341 raft_consensus.cc:2454] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dafc2cab4fc4235805a1126681d398b in term 0.
I20260620 22:55:43.703528 22405 leader_election.cc:302] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dafc2cab4fc4235805a1126681d398b, f3f51c131a6e477a899ff8992574f8af; no voters:
I20260620 22:55:43.703673 22540 raft_consensus.cc:2781] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 22:55:43.703732 22540 raft_consensus.cc:482] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 22:55:43.703754 22540 raft_consensus.cc:3037] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:43.704409 22540 raft_consensus.cc:504] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.704524 22540 leader_election.cc:288] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [CANDIDATE]: Term 1 election: Requested vote from peers f3f51c131a6e477a899ff8992574f8af (127.2.90.129:38745), d03b640fd9cc4ac6b45078ec7aafc2fe (127.2.90.130:35993)
I20260620 22:55:43.704658 22210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e21bb772ff484d85af80b2bbf731d7" candidate_uuid: "4dafc2cab4fc4235805a1126681d398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f51c131a6e477a899ff8992574f8af"
I20260620 22:55:43.704694 22341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27e21bb772ff484d85af80b2bbf731d7" candidate_uuid: "4dafc2cab4fc4235805a1126681d398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe"
I20260620 22:55:43.704739 22210 raft_consensus.cc:3037] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:43.704766 22341 raft_consensus.cc:3037] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:43.705431 22341 raft_consensus.cc:2454] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dafc2cab4fc4235805a1126681d398b in term 1.
I20260620 22:55:43.705458 22210 raft_consensus.cc:2454] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dafc2cab4fc4235805a1126681d398b in term 1.
I20260620 22:55:43.705579 22408 leader_election.cc:302] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dafc2cab4fc4235805a1126681d398b, d03b640fd9cc4ac6b45078ec7aafc2fe; no voters:
I20260620 22:55:43.705662 22540 raft_consensus.cc:2781] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:43.705807 22540 raft_consensus.cc:686] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 1 LEADER]: Becoming Leader. State: Replica: 4dafc2cab4fc4235805a1126681d398b, State: Running, Role: LEADER
I20260620 22:55:43.705891 22540 consensus_queue.cc:237] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } }
I20260620 22:55:43.706710 22068 catalog_manager.cc:5495] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dafc2cab4fc4235805a1126681d398b (127.2.90.131). New cstate: current_term: 1 leader_uuid: "4dafc2cab4fc4235805a1126681d398b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dafc2cab4fc4235805a1126681d398b" member_type: VOTER last_known_addr { host: "127.2.90.131" port: 44793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20260620 22:55:43.734232 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.731964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.2.90.131:44793)
0620 22:55:43.732041 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:43.732043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:43.732051 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:55:43.732087 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:43.732088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:43.732104 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:43.732164 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:55:43.732291 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.732798 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.733535 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:43.733731 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:43.733783 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:43.733925 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:55:43.734030 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:55:43.734101 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:43.734118 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 22:55:43.734137 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 22:55:43.734236 22388 tablet.cc:2369] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:55:43.735980 22341 raft_consensus.cc:1261] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 1 FOLLOWER]: Refusing update from remote peer 4dafc2cab4fc4235805a1126681d398b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:43.735980 22210 raft_consensus.cc:1261] T 27e21bb772ff484d85af80b2bbf731d7 P f3f51c131a6e477a899ff8992574f8af [term 1 FOLLOWER]: Refusing update from remote peer 4dafc2cab4fc4235805a1126681d398b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:55:43.736341 22542 consensus_queue.cc:1035] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f51c131a6e477a899ff8992574f8af" member_type: VOTER last_known_addr { host: "127.2.90.129" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:43.736465 22540 consensus_queue.cc:1035] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03b640fd9cc4ac6b45078ec7aafc2fe" member_type: VOTER last_known_addr { host: "127.2.90.130" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:55:43.740980 22067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:43.741063 22067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 22:55:43.741693 22067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa5fe264b324912a0fe75112e42e701: Sending DeleteTablet for 3 replicas of tablet 27e21bb772ff484d85af80b2bbf731d7
I20260620 22:55:43.742023 22452 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e21bb772ff484d85af80b2bbf731d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:43 UTC) from {username='slave'} at 127.0.0.1:51244
I20260620 22:55:43.742024 22321 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e21bb772ff484d85af80b2bbf731d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:43 UTC) from {username='slave'} at 127.0.0.1:43094
I20260620 22:55:43.742233 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet 27e21bb772ff484d85af80b2bbf731d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 22:55:43 UTC) from {username='slave'} at 127.0.0.1:58628
I20260620 22:55:43.742259 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22128
I20260620 22:55:43.742319 22556 tablet_replica.cc:330] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: stopping tablet replica
I20260620 22:55:43.742420 22556 raft_consensus.cc:2229] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 1 LEADER]: Raft consensus shutting down.
I20260620 22:55:43.742496 22556 raft_consensus.cc:2258] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:43.742961 22556 ts_tablet_manager.cc:1905] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:43.744520 22555 tablet_replica.cc:330] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: stopping tablet replica
I20260620 22:55:43.744626 22555 raft_consensus.cc:2229] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 22:55:43.744643 22556 ts_tablet_manager.cc:1918] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 22:55:43.744750 22555 raft_consensus.cc:2258] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 22:55:43.744840 22556 log.cc:1196] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Deleting WAL directory at /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781995912840861-2410-0/minicluster-data/ts-2/wal/wals/27e21bb772ff484d85af80b2bbf731d7
I20260620 22:55:43.745193 22556 ts_tablet_manager.cc:1939] T 27e21bb772ff484d85af80b2bbf731d7 P 4dafc2cab4fc4235805a1126681d398b: Deleting consensus metadata
I20260620 22:55:43.745561 22053 catalog_manager.cc:4841] TS 4dafc2cab4fc4235805a1126681d398b (127.2.90.131:44793): tablet 27e21bb772ff484d85af80b2bbf731d7 (table test-table [id=04bd64b300d4446f9d7dc05ff14a6a5b]) successfully deleted
W20260620 22:55:43.748051 22052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.90.129:38745 (error 108)
I20260620 22:55:43.748193 22555 ts_tablet_manager.cc:1905] T 27e21bb772ff484d85af80b2bbf731d7 P d03b640fd9cc4ac6b45078ec7aafc2fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 22:55:43.748267 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22259
I20260620 22:55:43.748828 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.748409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.129:38745 (local address 127.0.0.1:58638)
0620 22:55:43.748761 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.748801 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.129:38745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
W20260620 22:55:43.748960 22052 catalog_manager.cc:4584] TS f3f51c131a6e477a899ff8992574f8af (127.2.90.129:38745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27e21bb772ff484d85af80b2bbf731d7: Network error: Client connection negotiation failed: client connection to 127.2.90.129:38745: connect: Connection refused (error 111)
I20260620 22:55:43.753681 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22390
I20260620 22:55:43.753767 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:43.753654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.130:35993 (local address 127.0.0.1:43106)
0620 22:55:43.753702 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:55:43.753736 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.130:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 22:55:43.757308 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22037
2026-06-20T22:55:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (727 ms)
[----------] 6 tests from EncryptionPolicyTest (7690 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:43 dist-test-slave-4mtw krb5kdc[22560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:43 dist-test-slave-4mtw krb5kdc[22560](info): set up 2 sockets
Jun 20 22:55:43 dist-test-slave-4mtw krb5kdc[22560](info): commencing operation
krb5kdc: starting...
W20260620 22:55:44.696367 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:44 dist-test-slave-4mtw krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:44.802871 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:44.879966 22573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:44.880173 22573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:44.880195 22573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:44.881312 22573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:44.881346 22573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:44.881359 22573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:44.881371 22573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:44.881384 22573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:44.882783 22573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:44.882977 22573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:44.883149 22573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:44.887566 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:44.887566 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:44.887650 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:44.887732 22573 server_base.cc:1034] running on GCE node
I20260620 22:55:44.887984 22573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:44.888257 22573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:44.889387 22573 hybrid_clock.cc:648] HybridClock initialized: now 1781996144889371 us; error 28 us; skew 500 ppm
Jun 20 22:55:44 dist-test-slave-4mtw krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:44.891705 22573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:44.892125 22573 webserver.cc:460] Webserver started at http://127.0.0.1:42431/ using document root <none> and password file <none>
I20260620 22:55:44.892331 22573 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:44.892378 22573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:44.892499 22573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:44.893554 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "c6d81b3044974c869253604a0584c035"
format_stamp: "Formatted at 2026-06-20 22:55:44 on dist-test-slave-4mtw"
server_key: "641e612f14dc2a014bd45308cf78cf08"
server_key_iv: "680462c48a62024309607f0a7d7fe014"
server_key_version: "encryptionkey@0"
I20260620 22:55:44.893877 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "c6d81b3044974c869253604a0584c035"
format_stamp: "Formatted at 2026-06-20 22:55:44 on dist-test-slave-4mtw"
server_key: "641e612f14dc2a014bd45308cf78cf08"
server_key_iv: "680462c48a62024309607f0a7d7fe014"
server_key_version: "encryptionkey@0"
I20260620 22:55:44.895105 22573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:44.895818 22588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:44.895985 22573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:44.896067 22573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "c6d81b3044974c869253604a0584c035"
format_stamp: "Formatted at 2026-06-20 22:55:44 on dist-test-slave-4mtw"
server_key: "641e612f14dc2a014bd45308cf78cf08"
server_key_iv: "680462c48a62024309607f0a7d7fe014"
server_key_version: "encryptionkey@0"
I20260620 22:55:44.896133 22573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:44.904713 22573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:44.906658 22573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:44.906781 22573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:44.910422 22573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46059
I20260620 22:55:44.910451 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46059 every 8 connection(s)
I20260620 22:55:44.910754 22573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:44.911672 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:44.913897 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035: Bootstrap starting.
I20260620 22:55:44.914494 22641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:44.914757 22641 log.cc:824] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:44.915421 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035: No bootstrap required, opened a new log
I20260620 22:55:44.916785 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22573
I20260620 22:55:44.916903 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:44.917075 2410 external_mini_cluster.cc:1428] Setting key 4e344b053ef6002b61fe7922e552e522
I20260620 22:55:44.917500 22641 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d81b3044974c869253604a0584c035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260620 22:55:44.917644 22641 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:44.917675 22641 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d81b3044974c869253604a0584c035, State: Initialized, Role: FOLLOWER
I20260620 22:55:44.917806 22641 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d81b3044974c869253604a0584c035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260620 22:55:44.917878 22641 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:44.917908 22641 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:44.917958 22641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:44.918469 22641 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d81b3044974c869253604a0584c035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260620 22:55:44.918561 22641 leader_election.cc:302] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d81b3044974c869253604a0584c035; no voters:
I20260620 22:55:44.918718 22641 leader_election.cc:288] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:44.918767 22646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:44.918870 22646 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [term 1 LEADER]: Becoming Leader. State: Replica: c6d81b3044974c869253604a0584c035, State: Running, Role: LEADER
I20260620 22:55:44.918957 22641 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:44.918951 22646 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d81b3044974c869253604a0584c035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } }
I20260620 22:55:44.919435 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d81b3044974c869253604a0584c035. Latest consensus state: current_term: 1 leader_uuid: "c6d81b3044974c869253604a0584c035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d81b3044974c869253604a0584c035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } } }
I20260620 22:55:44.919516 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:44.919463 22647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d81b3044974c869253604a0584c035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d81b3044974c869253604a0584c035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46059 } } }
I20260620 22:55:44.919700 22647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:44.920229 22654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:44.920744 22654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:44.922101 22654 catalog_manager.cc:1347] Generated new cluster ID: b07b628d4cb74fbc98846ac2ea931e1f
I20260620 22:55:44.922150 22654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:55:44 dist-test-slave-4mtw krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:55:44.929033 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:44.917602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46911 (local address 127.0.0.1:46059)
0620 22:55:44.917834 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:44.917844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:44.917874 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:55:44.917976 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:44.917983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:44.918037 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:44.918160 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:44.918892 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:44.921351 (+ 2459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:44.922359 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:44.922940 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:44.923041 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:44.925531 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:44.925535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:44.925535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:44.925546 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:44.927429 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:44.927822 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:44.927825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:44.927826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:44.927941 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:44.928134 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:44.928135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:44.928136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:44.928454 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:44.928606 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:44.928724 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 22:55:44.928826 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":124,"threads_started":1}
I20260620 22:55:44.929947 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:44.934965 22654 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:44.935663 22654 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:44.938442 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:44.931266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39925 (local address 127.0.0.1:46059)
0620 22:55:44.931315 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:44.931316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:44.931518 (+ 202us) server_negotiation.cc:407] Connection header received
0620 22:55:44.931576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:44.931577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:44.931586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:44.931621 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:44.932218 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:44.932985 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:44.933988 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:44.934306 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:44.934367 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:44.935828 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:44.935829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:44.935829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:44.935836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:44.937301 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:44.937674 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:44.937675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:44.937675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:44.937723 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:44.937902 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:44.937902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:44.937902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:44.938018 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:44.938099 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:44.938301 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 22:55:44.938357 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 22:55:44.940204 22654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d81b3044974c869253604a0584c035: Generated new TSK 0
I20260620 22:55:44.940372 22654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:44.944304 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22573
W20260620 22:55:44.948347 22558 connection.cc:535] client connection to 127.0.0.1:46059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46059): Connection reset by peer (error 104)
2026-06-20T22:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:44 dist-test-slave-4mtw krb5kdc[22672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:44 dist-test-slave-4mtw krb5kdc[22672](info): set up 2 sockets
Jun 20 22:55:44 dist-test-slave-4mtw krb5kdc[22672](info): commencing operation
krb5kdc: starting...
W20260620 22:55:45.900669 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:45 dist-test-slave-4mtw krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:46.009152 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:46.088403 22685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:46.088625 22685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:46.088647 22685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:46.089774 22685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:46.089813 22685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:46.089828 22685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:46.089838 22685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:46.089850 22685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:46.091254 22685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:46.091437 22685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:46.091614 22685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:46.096153 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:46.096196 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:46.096161 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:46.096249 22685 server_base.cc:1034] running on GCE node
I20260620 22:55:46.096552 22685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:46.096858 22685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:46.097994 22685 hybrid_clock.cc:648] HybridClock initialized: now 1781996146097977 us; error 32 us; skew 500 ppm
Jun 20 22:55:46 dist-test-slave-4mtw krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:46.100250 22685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:46.100648 22685 webserver.cc:460] Webserver started at http://127.0.0.1:44491/ using document root <none> and password file <none>
I20260620 22:55:46.100878 22685 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:46.100924 22685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:46.101032 22685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:46.102020 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "81413ae353174aaca58a5342124ce7c2"
format_stamp: "Formatted at 2026-06-20 22:55:46 on dist-test-slave-4mtw"
server_key: "a65fdb1b221040a061b5f8be4b7e5eec"
server_key_iv: "c0124de105edb462570d250f3e484665"
server_key_version: "encryptionkey@0"
I20260620 22:55:46.102331 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "81413ae353174aaca58a5342124ce7c2"
format_stamp: "Formatted at 2026-06-20 22:55:46 on dist-test-slave-4mtw"
server_key: "a65fdb1b221040a061b5f8be4b7e5eec"
server_key_iv: "c0124de105edb462570d250f3e484665"
server_key_version: "encryptionkey@0"
I20260620 22:55:46.103646 22685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:55:46.104238 22700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:46.104367 22685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:46.104446 22685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "81413ae353174aaca58a5342124ce7c2"
format_stamp: "Formatted at 2026-06-20 22:55:46 on dist-test-slave-4mtw"
server_key: "a65fdb1b221040a061b5f8be4b7e5eec"
server_key_iv: "c0124de105edb462570d250f3e484665"
server_key_version: "encryptionkey@0"
I20260620 22:55:46.104516 22685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:46.115903 22685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:46.117733 22685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:46.117862 22685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:46.121515 22685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36927
I20260620 22:55:46.121548 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36927 every 8 connection(s)
I20260620 22:55:46.121873 22685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:46.122783 22753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:46.123167 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22685
I20260620 22:55:46.123267 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:46.123435 2410 external_mini_cluster.cc:1428] Setting key 8c75f131083a6a8a4b9fd294615474c6
I20260620 22:55:46.125775 22753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2: Bootstrap starting.
I20260620 22:55:46.126436 22753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:46.126694 22753 log.cc:824] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:46.127326 22753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2: No bootstrap required, opened a new log
I20260620 22:55:46.128840 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:46.123875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53175 (local address 127.0.0.1:36927)
0620 22:55:46.124031 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:46.124040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:46.124073 (+ 33us) server_negotiation.cc:407] Connection header received
0620 22:55:46.124556 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:46.124564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:46.124627 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:46.124799 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:46.124960 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:46.126250 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:46.127007 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:46.127456 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:46.127542 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:46.128191 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:46.128195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:46.128195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:46.128369 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:46.128400 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:46.128410 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:46.128471 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:46.128588 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:55:46.128610 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260620 22:55:46.129518 22753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81413ae353174aaca58a5342124ce7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260620 22:55:46.129663 22753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:46.129706 22753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81413ae353174aaca58a5342124ce7c2, State: Initialized, Role: FOLLOWER
I20260620 22:55:46.129720 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:46.129810 22753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81413ae353174aaca58a5342124ce7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260620 22:55:46.129863 22753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:46.129881 22753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:46.129906 22753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:46.130374 22753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81413ae353174aaca58a5342124ce7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260620 22:55:46.130445 22753 leader_election.cc:302] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81413ae353174aaca58a5342124ce7c2; no voters:
I20260620 22:55:46.130577 22753 leader_election.cc:288] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:46.130616 22760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:46.130748 22760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [term 1 LEADER]: Becoming Leader. State: Replica: 81413ae353174aaca58a5342124ce7c2, State: Running, Role: LEADER
I20260620 22:55:46.130800 22753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:46.130837 22760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81413ae353174aaca58a5342124ce7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260620 22:55:46.131197 22760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81413ae353174aaca58a5342124ce7c2. Latest consensus state: current_term: 1 leader_uuid: "81413ae353174aaca58a5342124ce7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81413ae353174aaca58a5342124ce7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260620 22:55:46.131203 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81413ae353174aaca58a5342124ce7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81413ae353174aaca58a5342124ce7c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260620 22:55:46.131283 22760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:46.131346 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:46.131804 22770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:46.132304 22770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:46.134438 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:46.130909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33997 (local address 127.0.0.1:36927)
0620 22:55:46.130942 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:46.130944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:46.131116 (+ 172us) server_negotiation.cc:407] Connection header received
0620 22:55:46.131205 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:46.131205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:46.131214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:46.131241 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:46.131530 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:46.132360 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:46.133283 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:46.133489 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:46.133554 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:46.134183 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:46.134184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:46.134184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:46.134188 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:46.134204 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:46.134208 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:46.134254 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:46.134362 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 22:55:46.134374 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:55:46.134881 22770 catalog_manager.cc:1347] Generated new cluster ID: 51c03bd40adf49dba47fd3e2657c92c5
I20260620 22:55:46.134927 22770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:46.149519 22770 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:46.150220 22770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:46.159087 22770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81413ae353174aaca58a5342124ce7c2: Generated new TSK 0
I20260620 22:55:46.159269 22770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:46.162866 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22685
W20260620 22:55:46.166507 22670 connection.cc:535] client connection to 127.0.0.1:36927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36927): Connection reset by peer (error 104)
2026-06-20T22:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:46 dist-test-slave-4mtw krb5kdc[22784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:46 dist-test-slave-4mtw krb5kdc[22784](info): set up 2 sockets
Jun 20 22:55:46 dist-test-slave-4mtw krb5kdc[22784](info): commencing operation
krb5kdc: starting...
W20260620 22:55:47.104413 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:47 dist-test-slave-4mtw krb5kdc[22784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:47.213032 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:47.290071 22797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:47.290271 22797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:47.290293 22797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:47.291429 22797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:47.291472 22797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:47.291486 22797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:47.291508 22797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:47.291530 22797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:47.293004 22797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:47.293231 22797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:47.293431 22797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:55:47.297850 22797 server_base.cc:1034] running on GCE node
W20260620 22:55:47.297802 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:47.297825 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:47.297803 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:47.298250 22797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:47.298523 22797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:47.299672 22797 hybrid_clock.cc:648] HybridClock initialized: now 1781996147299660 us; error 29 us; skew 500 ppm
Jun 20 22:55:47 dist-test-slave-4mtw krb5kdc[22784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:47.301894 22797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:47.302265 22797 webserver.cc:460] Webserver started at http://127.0.0.1:39861/ using document root <none> and password file <none>
I20260620 22:55:47.302441 22797 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:47.302486 22797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:47.302578 22797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:47.303562 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "256b96cea4c64d90a0eb76f4330b839f"
format_stamp: "Formatted at 2026-06-20 22:55:47 on dist-test-slave-4mtw"
server_key: "fe6cd317aa3d5b3ea3e26976fe5f29fa"
server_key_iv: "a0a2a33a2622d9535a6edfdf584dd024"
server_key_version: "encryptionkey@0"
I20260620 22:55:47.303864 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "256b96cea4c64d90a0eb76f4330b839f"
format_stamp: "Formatted at 2026-06-20 22:55:47 on dist-test-slave-4mtw"
server_key: "fe6cd317aa3d5b3ea3e26976fe5f29fa"
server_key_iv: "a0a2a33a2622d9535a6edfdf584dd024"
server_key_version: "encryptionkey@0"
I20260620 22:55:47.305053 22797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:55:47.305790 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:47.305958 22797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 22:55:47.306034 22797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "256b96cea4c64d90a0eb76f4330b839f"
format_stamp: "Formatted at 2026-06-20 22:55:47 on dist-test-slave-4mtw"
server_key: "fe6cd317aa3d5b3ea3e26976fe5f29fa"
server_key_iv: "a0a2a33a2622d9535a6edfdf584dd024"
server_key_version: "encryptionkey@0"
I20260620 22:55:47.306084 22797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:47.309073 22797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:47.310609 22797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:47.310703 22797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:47.314160 22797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45809
I20260620 22:55:47.314191 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45809 every 8 connection(s)
I20260620 22:55:47.314453 22797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:47.315330 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:47.316936 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22797
I20260620 22:55:47.317034 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:47.317179 2410 external_mini_cluster.cc:1428] Setting key d446f93d8017711489c8435cd47503d0
I20260620 22:55:47.317703 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f: Bootstrap starting.
I20260620 22:55:47.318372 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:47.318670 22865 log.cc:824] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:47.319480 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f: No bootstrap required, opened a new log
I20260620 22:55:47.319495 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:47.317620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50529 (local address 127.0.0.1:45809)
0620 22:55:47.317777 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:47.317786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:47.317818 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:55:47.318355 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:47.318364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:47.318399 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:47.318567 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:47.319108 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:47.319111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:47.319111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:47.319114 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:47.319145 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:47.319155 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:47.319237 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:47.319271 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:55:47.319309 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260620 22:55:47.320380 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:47.321638 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256b96cea4c64d90a0eb76f4330b839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260620 22:55:47.321813 22865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:47.321857 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256b96cea4c64d90a0eb76f4330b839f, State: Initialized, Role: FOLLOWER
I20260620 22:55:47.321983 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256b96cea4c64d90a0eb76f4330b839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260620 22:55:47.322077 22865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:47.322115 22865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:47.322165 22865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:47.322481 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:47.321562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59787 (local address 127.0.0.1:45809)
0620 22:55:47.321593 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:47.321595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:47.321716 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:55:47.321781 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:47.321781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:47.321784 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:47.321840 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:47.322330 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:47.322331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:47.322331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:47.322334 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:47.322345 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:47.322348 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:47.322382 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:47.322434 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 22:55:47.322443 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 22:55:47.322800 22865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256b96cea4c64d90a0eb76f4330b839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260620 22:55:47.322904 22865 leader_election.cc:302] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 256b96cea4c64d90a0eb76f4330b839f; no voters:
I20260620 22:55:47.323056 22865 leader_election.cc:288] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:47.323093 22875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:47.323201 22875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [term 1 LEADER]: Becoming Leader. State: Replica: 256b96cea4c64d90a0eb76f4330b839f, State: Running, Role: LEADER
I20260620 22:55:47.323272 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:47.323343 22875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256b96cea4c64d90a0eb76f4330b839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260620 22:55:47.323683 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "256b96cea4c64d90a0eb76f4330b839f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256b96cea4c64d90a0eb76f4330b839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } } }
I20260620 22:55:47.323752 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:47.323748 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 256b96cea4c64d90a0eb76f4330b839f. Latest consensus state: current_term: 1 leader_uuid: "256b96cea4c64d90a0eb76f4330b839f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256b96cea4c64d90a0eb76f4330b839f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } } }
I20260620 22:55:47.323861 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:47.324544 22884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:47.325102 22884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:47.326361 22884 catalog_manager.cc:1347] Generated new cluster ID: 1c55aee311ee4a929b7a82f7caf1af6d
I20260620 22:55:47.326411 22884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:47.335114 22884 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:47.335269 22884 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:47.345602 22884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 256b96cea4c64d90a0eb76f4330b839f: Generated new TSK 0
I20260620 22:55:47.345817 22884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:47.353494 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22797
2026-06-20T22:55:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:47 dist-test-slave-4mtw krb5kdc[22896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:47 dist-test-slave-4mtw krb5kdc[22896](info): set up 2 sockets
Jun 20 22:55:47 dist-test-slave-4mtw krb5kdc[22896](info): commencing operation
krb5kdc: starting...
W20260620 22:55:48.296288 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:48 dist-test-slave-4mtw krb5kdc[22896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:48.401078 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:48.477303 22909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:48.477523 22909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:48.477555 22909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:48.478690 22909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:48.478732 22909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:48.478750 22909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:48.478767 22909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:48.478782 22909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:48.480226 22909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:48.480468 22909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:48.480666 22909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:48.485049 22917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:48.485064 22914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:48.485147 22909 server_base.cc:1034] running on GCE node
W20260620 22:55:48.485065 22915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:48.485512 22909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:48.485790 22909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:48.486948 22909 hybrid_clock.cc:648] HybridClock initialized: now 1781996148486933 us; error 27 us; skew 500 ppm
Jun 20 22:55:48 dist-test-slave-4mtw krb5kdc[22896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:48.492056 22909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:48.492463 22909 webserver.cc:460] Webserver started at http://127.0.0.1:35243/ using document root <none> and password file <none>
I20260620 22:55:48.492666 22909 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:48.492749 22909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:48.492877 22909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:48.493872 22909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "955dfd7c3cb34438880055368e52ad82"
format_stamp: "Formatted at 2026-06-20 22:55:48 on dist-test-slave-4mtw"
server_key: "d0dc106c873b998b8d632202da4e0c7b"
server_key_iv: "a46ada44ab237dc61d3b911f44423ee5"
server_key_version: "encryptionkey@0"
I20260620 22:55:48.494205 22909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "955dfd7c3cb34438880055368e52ad82"
format_stamp: "Formatted at 2026-06-20 22:55:48 on dist-test-slave-4mtw"
server_key: "d0dc106c873b998b8d632202da4e0c7b"
server_key_iv: "a46ada44ab237dc61d3b911f44423ee5"
server_key_version: "encryptionkey@0"
I20260620 22:55:48.495397 22909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:48.496002 22924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:48.496150 22909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:48.496227 22909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "955dfd7c3cb34438880055368e52ad82"
format_stamp: "Formatted at 2026-06-20 22:55:48 on dist-test-slave-4mtw"
server_key: "d0dc106c873b998b8d632202da4e0c7b"
server_key_iv: "a46ada44ab237dc61d3b911f44423ee5"
server_key_version: "encryptionkey@0"
I20260620 22:55:48.496292 22909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:48.522464 22909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:48.524171 22909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:48.524308 22909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:48.527995 22909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37279
I20260620 22:55:48.528028 22976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37279 every 8 connection(s)
I20260620 22:55:48.528358 22909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:48.529280 22977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:48.531436 22977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82: Bootstrap starting.
I20260620 22:55:48.531975 22977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:48.532213 22977 log.cc:824] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:48.532868 22977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82: No bootstrap required, opened a new log
I20260620 22:55:48.534363 22977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955dfd7c3cb34438880055368e52ad82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37279 } }
I20260620 22:55:48.534507 22977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:48.534540 22977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955dfd7c3cb34438880055368e52ad82, State: Initialized, Role: FOLLOWER
I20260620 22:55:48.534680 22977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955dfd7c3cb34438880055368e52ad82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37279 } }
I20260620 22:55:48.534754 22977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:48.534786 22977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:48.534830 22977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:48.535126 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 22909
I20260620 22:55:48.535239 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:48.535405 2410 external_mini_cluster.cc:1428] Setting key faf63a46ad11b3a1a7490828f0642651
I20260620 22:55:48.535439 22977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955dfd7c3cb34438880055368e52ad82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37279 } }
I20260620 22:55:48.535547 22977 leader_election.cc:302] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 955dfd7c3cb34438880055368e52ad82; no voters:
I20260620 22:55:48.535696 22977 leader_election.cc:288] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:48.535722 22980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:48.536000 22977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:48.536093 22980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [term 1 LEADER]: Becoming Leader. State: Replica: 955dfd7c3cb34438880055368e52ad82, State: Running, Role: LEADER
I20260620 22:55:48.536196 22980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955dfd7c3cb34438880055368e52ad82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37279 } }
I20260620 22:55:48.536538 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955dfd7c3cb34438880055368e52ad82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955dfd7c3cb34438880055368e52ad82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37279 } } }
I20260620 22:55:48.536703 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:48.537052 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955dfd7c3cb34438880055368e52ad82. Latest consensus state: current_term: 1 leader_uuid: "955dfd7c3cb34438880055368e52ad82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955dfd7c3cb34438880055368e52ad82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37279 } } }
I20260620 22:55:48.537195 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:48.537305 22991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:48.537899 22991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:48.540138 22991 catalog_manager.cc:1347] Generated new cluster ID: 9c44def63aaf4095bba53adc771ef904
I20260620 22:55:48.540189 22991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 22:55:48 dist-test-slave-4mtw krb5kdc[22896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:55:48.546514 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:48.535850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42137 (local address 127.0.0.1:37279)
0620 22:55:48.536537 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:48.536546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:48.536578 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:55:48.536719 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:48.536727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:48.536788 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:48.536982 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:48.537825 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:48.539034 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:48.539761 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:48.540357 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:48.540454 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:48.543060 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:48.543064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:48.543064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:48.543076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:48.544965 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:48.545320 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:48.545323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:48.545323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:48.545440 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:48.545609 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:48.545609 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:48.545610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:48.545928 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:48.546079 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:48.546172 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:55:48.546307 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":58,"threads_started":1}
I20260620 22:55:48.547351 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:48.551283 22991 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:48.551860 22991 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:48.553829 22991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955dfd7c3cb34438880055368e52ad82: Generated new TSK 0
I20260620 22:55:48.554234 22991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:48.555372 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:48.548673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34555 (local address 127.0.0.1:37279)
0620 22:55:48.548740 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:48.548742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:48.548907 (+ 165us) server_negotiation.cc:407] Connection header received
0620 22:55:48.548970 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:48.548971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:48.548982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:48.549029 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:48.549592 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:48.550299 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:48.551318 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:48.551620 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:48.551685 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:48.552944 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:48.552946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:48.552946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:48.552953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:48.554416 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:48.554769 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:48.554770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:48.554770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:48.554814 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:48.554981 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:48.554982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:48.554982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:48.555065 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:48.555129 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:48.555255 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 22:55:48.555318 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:55:48.556977 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 22909
2026-06-20T22:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:48 dist-test-slave-4mtw krb5kdc[23008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:48 dist-test-slave-4mtw krb5kdc[23008](info): set up 2 sockets
Jun 20 22:55:48 dist-test-slave-4mtw krb5kdc[23008](info): commencing operation
krb5kdc: starting...
W20260620 22:55:49.508348 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:49 dist-test-slave-4mtw krb5kdc[23008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:49.616508 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:49.694445 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:49.694645 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:49.694674 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:49.695802 23021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:49.695839 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:49.695858 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:49.695873 23021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:49.695888 23021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:49.697374 23021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:49.697599 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:49.697793 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:49.702131 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:49.702143 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:49.702139 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:49.702211 23021 server_base.cc:1034] running on GCE node
I20260620 22:55:49.702595 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:49.702845 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:49.703981 23021 hybrid_clock.cc:648] HybridClock initialized: now 1781996149703963 us; error 29 us; skew 500 ppm
Jun 20 22:55:49 dist-test-slave-4mtw krb5kdc[23008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:49.706322 23021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:49.706705 23021 webserver.cc:460] Webserver started at http://127.0.0.1:35319/ using document root <none> and password file <none>
I20260620 22:55:49.706911 23021 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:49.706957 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:49.707077 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:49.708043 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "65e2817586fd44cc8c9567e3db102888"
format_stamp: "Formatted at 2026-06-20 22:55:49 on dist-test-slave-4mtw"
server_key: "31b78dc7ecbbd9f6c6da6b53a1fa8611"
server_key_iv: "37383ef21ec05c52454e685838ce4b54"
server_key_version: "encryptionkey@0"
I20260620 22:55:49.708362 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "65e2817586fd44cc8c9567e3db102888"
format_stamp: "Formatted at 2026-06-20 22:55:49 on dist-test-slave-4mtw"
server_key: "31b78dc7ecbbd9f6c6da6b53a1fa8611"
server_key_iv: "37383ef21ec05c52454e685838ce4b54"
server_key_version: "encryptionkey@0"
I20260620 22:55:49.709652 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:55:49.710310 23036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:49.710459 23021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:49.710548 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "65e2817586fd44cc8c9567e3db102888"
format_stamp: "Formatted at 2026-06-20 22:55:49 on dist-test-slave-4mtw"
server_key: "31b78dc7ecbbd9f6c6da6b53a1fa8611"
server_key_iv: "37383ef21ec05c52454e685838ce4b54"
server_key_version: "encryptionkey@0"
I20260620 22:55:49.710608 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:49.717379 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:49.719445 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:49.719571 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:49.723127 23021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43023
I20260620 22:55:49.723156 23088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43023 every 8 connection(s)
I20260620 22:55:49.723444 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:49.724290 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:49.726372 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888: Bootstrap starting.
I20260620 22:55:49.726863 23089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:49.727077 23089 log.cc:824] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:49.727653 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888: No bootstrap required, opened a new log
I20260620 22:55:49.729151 23089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2817586fd44cc8c9567e3db102888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260620 22:55:49.729278 23089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:49.729298 23089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2817586fd44cc8c9567e3db102888, State: Initialized, Role: FOLLOWER
I20260620 22:55:49.729404 23089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2817586fd44cc8c9567e3db102888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260620 22:55:49.729462 23089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:49.729481 23089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:49.729511 23089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:49.729924 23089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2817586fd44cc8c9567e3db102888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260620 22:55:49.730001 23089 leader_election.cc:302] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e2817586fd44cc8c9567e3db102888; no voters:
I20260620 22:55:49.730115 23089 leader_election.cc:288] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:49.730162 23092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:49.730326 23089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:49.730340 23092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [term 1 LEADER]: Becoming Leader. State: Replica: 65e2817586fd44cc8c9567e3db102888, State: Running, Role: LEADER
I20260620 22:55:49.730444 23092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2817586fd44cc8c9567e3db102888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260620 22:55:49.730499 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23021
I20260620 22:55:49.730605 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:49.730758 2410 external_mini_cluster.cc:1428] Setting key 1b9da7edc691f3dcecf041798bd0ac3b
I20260620 22:55:49.731460 23093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65e2817586fd44cc8c9567e3db102888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2817586fd44cc8c9567e3db102888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } } }
I20260620 22:55:49.731551 23093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:49.731752 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e2817586fd44cc8c9567e3db102888. Latest consensus state: current_term: 1 leader_uuid: "65e2817586fd44cc8c9567e3db102888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e2817586fd44cc8c9567e3db102888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } } }
I20260620 22:55:49.731875 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888 [sys.catalog]: This master's current role is: LEADER
W20260620 22:55:49.732028 23108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:55:49.732079 23108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:55:49.732189 23109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:49.732362 23109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:49.733824 23109 catalog_manager.cc:1347] Generated new cluster ID: f09e3b67903b4c7da49aeee3b722310a
I20260620 22:55:49.733875 23109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:49.736878 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:49.731173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52017 (local address 127.0.0.1:43023)
0620 22:55:49.731370 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:49.731379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:49.731415 (+ 36us) server_negotiation.cc:407] Connection header received
0620 22:55:49.731586 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:49.731595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:49.731704 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:49.731916 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:49.732398 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:49.733611 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:49.734644 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:49.735240 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:49.735596 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:49.735955 (+ 359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:49.735958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:49.735958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:49.736142 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:49.736173 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:49.736186 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:49.736434 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:49.736463 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 22:55:49.736518 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260620 22:55:49.737634 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:49.742352 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:49.739105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49607 (local address 127.0.0.1:43023)
0620 22:55:49.739132 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:49.739134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:49.739250 (+ 116us) server_negotiation.cc:407] Connection header received
0620 22:55:49.739309 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:49.739309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:49.739317 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:49.739366 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:49.739500 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:49.740214 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:49.741243 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:49.741521 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:49.741572 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:49.742151 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:49.742152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:49.742152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:49.742156 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:49.742169 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:49.742172 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:49.742212 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:49.742263 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 22:55:49.742284 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 22:55:49.753396 23109 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:49.753924 23109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:49.760810 23109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e2817586fd44cc8c9567e3db102888: Generated new TSK 0
I20260620 22:55:49.760967 23109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:49.769696 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23021
2026-06-20T22:55:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:49 dist-test-slave-4mtw krb5kdc[23120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:49 dist-test-slave-4mtw krb5kdc[23120](info): set up 2 sockets
Jun 20 22:55:49 dist-test-slave-4mtw krb5kdc[23120](info): commencing operation
krb5kdc: starting...
W20260620 22:55:50.704475 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:50 dist-test-slave-4mtw krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:50.809109 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:50.885788 23133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:50.885998 23133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:50.886029 23133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:50.887151 23133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:50.887192 23133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:50.887210 23133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:50.887226 23133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:50.887243 23133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:50.888712 23133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:50.888947 23133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:50.889142 23133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:50.893563 23139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:50.893613 23138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:50.893646 23133 server_base.cc:1034] running on GCE node
W20260620 22:55:50.893570 23141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:50.893980 23133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:50.894259 23133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:50.895398 23133 hybrid_clock.cc:648] HybridClock initialized: now 1781996150895385 us; error 27 us; skew 500 ppm
Jun 20 22:55:50 dist-test-slave-4mtw krb5kdc[23120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:50.897655 23133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:50.898054 23133 webserver.cc:460] Webserver started at http://127.0.0.1:44647/ using document root <none> and password file <none>
I20260620 22:55:50.898248 23133 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:50.898295 23133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:50.898416 23133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:50.899422 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "0d9b59eb2ba04d308b536d1dea5c8977"
format_stamp: "Formatted at 2026-06-20 22:55:50 on dist-test-slave-4mtw"
server_key: "3b0d0e8ddd10ef9c722c98c26db5f268"
server_key_iv: "ddb047dd600420ead5baaea807331cfe"
server_key_version: "encryptionkey@0"
I20260620 22:55:50.899760 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "0d9b59eb2ba04d308b536d1dea5c8977"
format_stamp: "Formatted at 2026-06-20 22:55:50 on dist-test-slave-4mtw"
server_key: "3b0d0e8ddd10ef9c722c98c26db5f268"
server_key_iv: "ddb047dd600420ead5baaea807331cfe"
server_key_version: "encryptionkey@0"
I20260620 22:55:50.900997 23133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:55:50.901608 23148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:50.901752 23133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:55:50.901832 23133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "0d9b59eb2ba04d308b536d1dea5c8977"
format_stamp: "Formatted at 2026-06-20 22:55:50 on dist-test-slave-4mtw"
server_key: "3b0d0e8ddd10ef9c722c98c26db5f268"
server_key_iv: "ddb047dd600420ead5baaea807331cfe"
server_key_version: "encryptionkey@0"
I20260620 22:55:50.901889 23133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:50.905021 23133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:50.906723 23133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:50.906843 23133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:50.910439 23133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35063
I20260620 22:55:50.910478 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35063 every 8 connection(s)
I20260620 22:55:50.910768 23133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:50.911612 23201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:50.912883 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23133
I20260620 22:55:50.913002 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:50.913179 2410 external_mini_cluster.cc:1428] Setting key 112724a7f73ac5b65806b2e8479fd842
I20260620 22:55:50.914121 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977: Bootstrap starting.
I20260620 22:55:50.914736 23201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:50.914996 23201 log.cc:824] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:50.915446 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:50.913663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36411 (local address 127.0.0.1:35063)
0620 22:55:50.913823 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:50.913830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:50.913855 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:55:50.914287 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:50.914294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:50.914321 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:50.914437 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:50.915059 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:50.915062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:50.915063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:50.915067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:50.915110 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:50.915124 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:50.915197 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:50.915232 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:55:50.915270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260620 22:55:50.915657 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977: No bootstrap required, opened a new log
I20260620 22:55:50.916289 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:50.917632 23201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260620 22:55:50.917793 23201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:50.917841 23201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9b59eb2ba04d308b536d1dea5c8977, State: Initialized, Role: FOLLOWER
I20260620 22:55:50.917975 23201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260620 22:55:50.918069 23201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:50.918144 23201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:50.918205 23201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:50.918504 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:50.917548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54555 (local address 127.0.0.1:35063)
0620 22:55:50.917584 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:50.917586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:50.917690 (+ 104us) server_negotiation.cc:407] Connection header received
0620 22:55:50.917717 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:50.917718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:50.917722 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:50.917785 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:50.918317 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:50.918318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:50.918318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:50.918323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:50.918335 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:50.918339 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:50.918396 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:50.918431 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:55:50.918443 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:55:50.918792 23201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260620 22:55:50.918879 23201 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d9b59eb2ba04d308b536d1dea5c8977; no voters:
I20260620 22:55:50.919029 23201 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:50.919057 23211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:50.919174 23211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9b59eb2ba04d308b536d1dea5c8977, State: Running, Role: LEADER
I20260620 22:55:50.919247 23201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:50.919291 23211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } }
I20260620 22:55:50.919642 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } } }
I20260620 22:55:50.919654 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9b59eb2ba04d308b536d1dea5c8977. Latest consensus state: current_term: 1 leader_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9b59eb2ba04d308b536d1dea5c8977" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35063 } } }
I20260620 22:55:50.919821 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:50.919826 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:50.920181 23219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:50.920622 23219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:50.921963 23219 catalog_manager.cc:1347] Generated new cluster ID: 0185706a03274415892343eb8dbbcfbc
I20260620 22:55:50.922001 23219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:50.930644 23219 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:50.930785 23219 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:50.941712 23219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d9b59eb2ba04d308b536d1dea5c8977: Generated new TSK 0
I20260620 22:55:50.941905 23219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:50.949621 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23133
2026-06-20T22:55:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1180 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:50 dist-test-slave-4mtw krb5kdc[23232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:50 dist-test-slave-4mtw krb5kdc[23232](info): set up 2 sockets
Jun 20 22:55:50 dist-test-slave-4mtw krb5kdc[23232](info): commencing operation
krb5kdc: starting...
W20260620 22:55:51.884474 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:51 dist-test-slave-4mtw krb5kdc[23232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:51.989559 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:52.067886 23245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:52.068114 23245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:52.068136 23245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:52.069335 23245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:52.069381 23245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:52.069394 23245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:52.069418 23245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:52.069442 23245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:52.070928 23245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:52.071166 23245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:52.071369 23245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:52.075975 23251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:52.076016 23253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:52.075986 23250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:52.076138 23245 server_base.cc:1034] running on GCE node
I20260620 22:55:52.076455 23245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:52.076756 23245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:52.077919 23245 hybrid_clock.cc:648] HybridClock initialized: now 1781996152077895 us; error 33 us; skew 500 ppm
Jun 20 22:55:52 dist-test-slave-4mtw krb5kdc[23232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:52.080214 23245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:52.080614 23245 webserver.cc:460] Webserver started at http://127.0.0.1:39969/ using document root <none> and password file <none>
I20260620 22:55:52.080852 23245 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:52.080902 23245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:52.081027 23245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:52.082027 23245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "45f31d1e43d94814a3c24ece7166d9b8"
format_stamp: "Formatted at 2026-06-20 22:55:52 on dist-test-slave-4mtw"
server_key: "1dd0a8893abcf06a0ec5c764aa9956d1"
server_key_iv: "af2f1ea72a2a01ee7e0e8e0dfc475dba"
server_key_version: "encryptionkey@0"
I20260620 22:55:52.082345 23245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "45f31d1e43d94814a3c24ece7166d9b8"
format_stamp: "Formatted at 2026-06-20 22:55:52 on dist-test-slave-4mtw"
server_key: "1dd0a8893abcf06a0ec5c764aa9956d1"
server_key_iv: "af2f1ea72a2a01ee7e0e8e0dfc475dba"
server_key_version: "encryptionkey@0"
I20260620 22:55:52.083654 23245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:52.084273 23260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:52.084419 23245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:52.084497 23245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "45f31d1e43d94814a3c24ece7166d9b8"
format_stamp: "Formatted at 2026-06-20 22:55:52 on dist-test-slave-4mtw"
server_key: "1dd0a8893abcf06a0ec5c764aa9956d1"
server_key_iv: "af2f1ea72a2a01ee7e0e8e0dfc475dba"
server_key_version: "encryptionkey@0"
I20260620 22:55:52.084556 23245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:52.096796 23245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:52.098356 23245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:52.098479 23245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:52.102020 23245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39419
I20260620 22:55:52.102051 23312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39419 every 8 connection(s)
I20260620 22:55:52.102356 23245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:52.103256 23313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:52.103463 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23245
I20260620 22:55:52.103569 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:52.103729 2410 external_mini_cluster.cc:1428] Setting key 37fa82a31096da4024efed4e80b37cfb
I20260620 22:55:52.105659 23313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8: Bootstrap starting.
I20260620 22:55:52.106180 23313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:52.106433 23313 log.cc:824] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:52.107024 23313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8: No bootstrap required, opened a new log
I20260620 22:55:52.108713 23313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f31d1e43d94814a3c24ece7166d9b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 22:55:52.108852 23313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:52.108884 23313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f31d1e43d94814a3c24ece7166d9b8, State: Initialized, Role: FOLLOWER
I20260620 22:55:52.109009 23313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f31d1e43d94814a3c24ece7166d9b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 22:55:52.109082 23313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:52.109120 23313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:52.109167 23313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 22:55:52 dist-test-slave-4mtw krb5kdc[23232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:55:52.109800 23313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f31d1e43d94814a3c24ece7166d9b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 22:55:52.109937 23313 leader_election.cc:302] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f31d1e43d94814a3c24ece7166d9b8; no voters:
I20260620 22:55:52.110095 23313 leader_election.cc:288] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:52.110136 23318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:52.110316 23318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [term 1 LEADER]: Becoming Leader. State: Replica: 45f31d1e43d94814a3c24ece7166d9b8, State: Running, Role: LEADER
I20260620 22:55:52.110320 23313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:52.110473 23318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f31d1e43d94814a3c24ece7166d9b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } }
I20260620 22:55:52.110867 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f31d1e43d94814a3c24ece7166d9b8. Latest consensus state: current_term: 1 leader_uuid: "45f31d1e43d94814a3c24ece7166d9b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f31d1e43d94814a3c24ece7166d9b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } } }
I20260620 22:55:52.110942 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:52.111222 23327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:52.111097 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f31d1e43d94814a3c24ece7166d9b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f31d1e43d94814a3c24ece7166d9b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39419 } } }
I20260620 22:55:52.111395 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:52.111680 23327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:52.113402 23327 catalog_manager.cc:1347] Generated new cluster ID: 777b69ccd3744835b5b6fc868ac54d68
I20260620 22:55:52.113447 23327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:52.113929 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:52.104197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:59835 (local address 127.0.0.1:39419)
0620 22:55:52.104388 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:52.104396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:52.104421 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:55:52.104873 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:52.104880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:52.104933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:52.105057 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:52.105646 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:52.106610 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:52.107378 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:52.107807 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:52.107905 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:52.110346 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:52.110350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:52.110351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:52.110363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:52.112350 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:52.112845 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:52.112848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:52.112849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:52.112950 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:52.113197 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:52.113197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:52.113198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:52.113428 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:52.113556 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:52.113683 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 22:55:52.113775 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260620 22:55:52.114744 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:52.121168 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:52.115890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:42815 (local address 127.0.0.1:39419)
0620 22:55:52.116041 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:52.116042 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:52.116051 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:55:52.116069 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:52.116069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:52.116078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:52.116134 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:52.116585 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:52.117093 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:52.117734 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:52.117974 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:52.118025 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:52.119018 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:52.119019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:52.119019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:52.119025 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:52.120164 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:52.120481 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:52.120482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:52.120482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:52.120530 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:52.120718 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:52.120719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:52.120719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:52.120826 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:52.120921 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:52.121038 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 22:55:52.121090 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":132}
I20260620 22:55:52.133935 23327 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:52.134407 23327 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:52.137504 23327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f31d1e43d94814a3c24ece7166d9b8: Generated new TSK 0
I20260620 22:55:52.137650 23327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:52.139367 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23245
W20260620 22:55:52.143226 23230 connection.cc:535] client connection to 127.0.0.1:39419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39419): Connection reset by peer (error 104)
2026-06-20T22:55:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1189 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:52 dist-test-slave-4mtw krb5kdc[23344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:52 dist-test-slave-4mtw krb5kdc[23344](info): set up 2 sockets
Jun 20 22:55:52 dist-test-slave-4mtw krb5kdc[23344](info): commencing operation
krb5kdc: starting...
W20260620 22:55:53.092412 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:53 dist-test-slave-4mtw krb5kdc[23344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:53.200067 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:53.275959 23357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:53.276151 23357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:53.276171 23357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:53.277269 23357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:53.277305 23357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:53.277318 23357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:53.277328 23357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:53.277338 23357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:53.278743 23357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:53.278927 23357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:53.279093 23357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:53.283524 23363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:53.283520 23362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:53.283555 23357 server_base.cc:1034] running on GCE node
W20260620 22:55:53.283525 23365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:53.283910 23357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:53.284179 23357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:53.285310 23357 hybrid_clock.cc:648] HybridClock initialized: now 1781996153285297 us; error 26 us; skew 500 ppm
Jun 20 22:55:53 dist-test-slave-4mtw krb5kdc[23344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:53.287514 23357 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:53.287873 23357 webserver.cc:460] Webserver started at http://127.0.0.1:42203/ using document root <none> and password file <none>
I20260620 22:55:53.288055 23357 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:53.288094 23357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:53.288177 23357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:53.289170 23357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "5113311fb0ec479dbdf0cec2881fc4a1"
format_stamp: "Formatted at 2026-06-20 22:55:53 on dist-test-slave-4mtw"
server_key: "dcd23de2d3d06bc640f62a07709c3f00"
server_key_iv: "e88134ec726951e6955cbfc55cbc4e44"
server_key_version: "encryptionkey@0"
I20260620 22:55:53.289444 23357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "5113311fb0ec479dbdf0cec2881fc4a1"
format_stamp: "Formatted at 2026-06-20 22:55:53 on dist-test-slave-4mtw"
server_key: "dcd23de2d3d06bc640f62a07709c3f00"
server_key_iv: "e88134ec726951e6955cbfc55cbc4e44"
server_key_version: "encryptionkey@0"
I20260620 22:55:53.290572 23357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:55:53.291162 23372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:53.291307 23357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:53.291380 23357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "5113311fb0ec479dbdf0cec2881fc4a1"
format_stamp: "Formatted at 2026-06-20 22:55:53 on dist-test-slave-4mtw"
server_key: "dcd23de2d3d06bc640f62a07709c3f00"
server_key_iv: "e88134ec726951e6955cbfc55cbc4e44"
server_key_version: "encryptionkey@0"
I20260620 22:55:53.291437 23357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:53.316797 23357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:53.318537 23357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:53.318643 23357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:53.322170 23357 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39903
I20260620 22:55:53.322224 23424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39903 every 8 connection(s)
I20260620 22:55:53.322531 23357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:53.323424 23425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:53.323684 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23357
I20260620 22:55:53.323803 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:53.324011 2410 external_mini_cluster.cc:1428] Setting key f6f817c8f9fa41ec6adc002d5ab6152a
I20260620 22:55:53.325929 23425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1: Bootstrap starting.
I20260620 22:55:53.326467 23425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:53.326705 23425 log.cc:824] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:53.327284 23425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1: No bootstrap required, opened a new log
I20260620 22:55:53.328982 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:53.324494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:60917 (local address 127.0.0.1:39903)
0620 22:55:53.324698 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:53.324708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:53.324739 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:55:53.325148 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:53.325154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:53.325203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:53.325333 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:53.325537 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:53.326448 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:53.327232 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:53.327656 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:53.327766 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:53.328408 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:53.328413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:53.328413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:53.328561 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:53.328589 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:53.328600 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:53.328664 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:53.328796 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 22:55:53.328823 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260620 22:55:53.329221 23425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 22:55:53.329375 23425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:53.329411 23425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5113311fb0ec479dbdf0cec2881fc4a1, State: Initialized, Role: FOLLOWER
I20260620 22:55:53.329531 23425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 22:55:53.329607 23425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:53.329641 23425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:53.329686 23425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:53.329754 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:53.330186 23425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 22:55:53.330277 23425 leader_election.cc:302] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5113311fb0ec479dbdf0cec2881fc4a1; no voters:
I20260620 22:55:53.330447 23425 leader_election.cc:288] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:53.330475 23430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:53.330588 23430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [term 1 LEADER]: Becoming Leader. State: Replica: 5113311fb0ec479dbdf0cec2881fc4a1, State: Running, Role: LEADER
I20260620 22:55:53.330683 23425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:53.330709 23430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } }
I20260620 22:55:53.331079 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5113311fb0ec479dbdf0cec2881fc4a1. Latest consensus state: current_term: 1 leader_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } } }
I20260620 22:55:53.331147 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:53.331110 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5113311fb0ec479dbdf0cec2881fc4a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39903 } } }
I20260620 22:55:53.331210 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1 [sys.catalog]: This master's current role is: LEADER
W20260620 22:55:53.332394 23451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:55:53.332448 23451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:55:53.332491 23439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:53.332638 23439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:53.334013 23439 catalog_manager.cc:1347] Generated new cluster ID: afa2ea9484864594be9a5b1c4ea95682
I20260620 22:55:53.334066 23439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:53.335420 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:53.331013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:43459 (local address 127.0.0.1:39903)
0620 22:55:53.331049 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:53.331050 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:53.332259 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 22:55:53.332338 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:53.332339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:53.332349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:53.332406 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:53.332576 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:53.333307 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:53.334128 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:53.334410 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:53.334458 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:53.335122 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:53.335123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:53.335124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:53.335127 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:53.335140 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:53.335143 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:53.335180 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:53.335347 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 22:55:53.335355 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:55:53.352363 23439 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:53.353148 23439 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:53.362660 23439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5113311fb0ec479dbdf0cec2881fc4a1: Generated new TSK 0
I20260620 22:55:53.362854 23439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:53.372074 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23357
W20260620 22:55:53.375773 23342 connection.cc:535] client connection to 127.0.0.1:39903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39903): Connection reset by peer (error 104)
2026-06-20T22:55:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:53 dist-test-slave-4mtw krb5kdc[23456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:53 dist-test-slave-4mtw krb5kdc[23456](info): set up 2 sockets
Jun 20 22:55:53 dist-test-slave-4mtw krb5kdc[23456](info): commencing operation
krb5kdc: starting...
W20260620 22:55:54.320411 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:54 dist-test-slave-4mtw krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:54.424955 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:54.502157 23469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:54.502360 23469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:54.502390 23469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:54.503528 23469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:54.503568 23469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:54.503587 23469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:54.503603 23469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:54.503619 23469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:54.505081 23469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:54.505314 23469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:54.505524 23469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:54.509917 23474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:54.509944 23477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:54.510329 23475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:54.510658 23469 server_base.cc:1034] running on GCE node
I20260620 22:55:54.510883 23469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:54.511143 23469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:54.512285 23469 hybrid_clock.cc:648] HybridClock initialized: now 1781996154512271 us; error 28 us; skew 500 ppm
Jun 20 22:55:54 dist-test-slave-4mtw krb5kdc[23456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:54.518220 23469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:54.518671 23469 webserver.cc:460] Webserver started at http://127.0.0.1:39043/ using document root <none> and password file <none>
I20260620 22:55:54.518882 23469 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:54.518930 23469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:54.519078 23469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:54.520090 23469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "bdd27391433242d38d75babbc80383e9"
format_stamp: "Formatted at 2026-06-20 22:55:54 on dist-test-slave-4mtw"
server_key: "213f0c3a21d64bc4f5a8dd9ec354bd27"
server_key_iv: "d0f05bde8d0185b6316493057f40f52d"
server_key_version: "encryptionkey@0"
I20260620 22:55:54.520421 23469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "bdd27391433242d38d75babbc80383e9"
format_stamp: "Formatted at 2026-06-20 22:55:54 on dist-test-slave-4mtw"
server_key: "213f0c3a21d64bc4f5a8dd9ec354bd27"
server_key_iv: "d0f05bde8d0185b6316493057f40f52d"
server_key_version: "encryptionkey@0"
I20260620 22:55:54.521683 23469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:54.522333 23484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:54.522487 23469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:54.522569 23469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "bdd27391433242d38d75babbc80383e9"
format_stamp: "Formatted at 2026-06-20 22:55:54 on dist-test-slave-4mtw"
server_key: "213f0c3a21d64bc4f5a8dd9ec354bd27"
server_key_iv: "d0f05bde8d0185b6316493057f40f52d"
server_key_version: "encryptionkey@0"
I20260620 22:55:54.522637 23469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:54.525856 23469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:54.527875 23469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:54.528009 23469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:54.531643 23469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45527
I20260620 22:55:54.531670 23536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45527 every 8 connection(s)
I20260620 22:55:54.531976 23469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:54.532893 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:54.534974 23537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9: Bootstrap starting.
I20260620 22:55:54.535470 23537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:54.535681 23537 log.cc:824] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:54.536249 23537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9: No bootstrap required, opened a new log
I20260620 22:55:54.537788 23537 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd27391433242d38d75babbc80383e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 22:55:54.537917 23537 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:54.537941 23537 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd27391433242d38d75babbc80383e9, State: Initialized, Role: FOLLOWER
I20260620 22:55:54.538050 23537 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd27391433242d38d75babbc80383e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 22:55:54.538120 23537 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:54.538146 23537 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:54.538178 23537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:54.538424 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23469
I20260620 22:55:54.538535 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:54.538710 2410 external_mini_cluster.cc:1428] Setting key 0b1526100bfc61eedf82f7b4e97e970d
I20260620 22:55:54.538792 23537 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd27391433242d38d75babbc80383e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 22:55:54.538890 23537 leader_election.cc:302] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdd27391433242d38d75babbc80383e9; no voters:
I20260620 22:55:54.539006 23537 leader_election.cc:288] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:54.539058 23540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:54.539199 23540 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [term 1 LEADER]: Becoming Leader. State: Replica: bdd27391433242d38d75babbc80383e9, State: Running, Role: LEADER
I20260620 22:55:54.539228 23537 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:54.539304 23540 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd27391433242d38d75babbc80383e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } }
I20260620 22:55:54.539717 23543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd27391433242d38d75babbc80383e9. Latest consensus state: current_term: 1 leader_uuid: "bdd27391433242d38d75babbc80383e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd27391433242d38d75babbc80383e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } } }
I20260620 22:55:54.539794 23543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:54.539992 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdd27391433242d38d75babbc80383e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd27391433242d38d75babbc80383e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45527 } } }
I20260620 22:55:54.540050 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:54.540449 23546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:54.541093 23546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:54.541219 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:54.539150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:41969 (local address 127.0.0.1:45527)
0620 22:55:54.539440 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:54.539449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:54.539525 (+ 76us) server_negotiation.cc:407] Connection header received
0620 22:55:54.539645 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:54.539653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:54.539687 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:54.540114 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:54.540623 (+ 509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:54.540627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:54.540628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:54.540632 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:54.540674 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:54.540712 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:54.540803 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:54.540961 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 22:55:54.541007 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":66,"threads_started":1}
I20260620 22:55:54.542387 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:54.544528 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:54.543656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:58231 (local address 127.0.0.1:45527)
0620 22:55:54.543698 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:54.543700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:54.543802 (+ 102us) server_negotiation.cc:407] Connection header received
0620 22:55:54.543870 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:54.543871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:54.543874 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:54.543918 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:54.544376 (+ 458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:54.544377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:54.544378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:54.544381 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:54.544391 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:54.544395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:54.544431 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:54.544477 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 22:55:54.544488 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:55:54.545028 23546 catalog_manager.cc:1347] Generated new cluster ID: 49909b650a934499915745f1e50fa273
I20260620 22:55:54.545076 23546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:54.551484 23546 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:54.551635 23546 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:54.555128 23546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdd27391433242d38d75babbc80383e9: Generated new TSK 0
I20260620 22:55:54.555289 23546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:54.558455 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23469
2026-06-20T22:55:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:54 dist-test-slave-4mtw krb5kdc[23568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:54 dist-test-slave-4mtw krb5kdc[23568](info): set up 2 sockets
Jun 20 22:55:54 dist-test-slave-4mtw krb5kdc[23568](info): commencing operation
krb5kdc: starting...
W20260620 22:55:55.504336 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:55 dist-test-slave-4mtw krb5kdc[23568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:55.611807 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:55.689992 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:55.690191 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:55.690212 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:55.691290 23581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:55.691322 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:55.691334 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:55.691346 23581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:55.691359 23581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:55.692795 23581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:55.692986 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:55.693153 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:55.697551 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:55.697572 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:55.697554 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:55.697628 23581 server_base.cc:1034] running on GCE node
I20260620 22:55:55.697994 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:55.698226 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:55.699358 23581 hybrid_clock.cc:648] HybridClock initialized: now 1781996155699345 us; error 30 us; skew 500 ppm
Jun 20 22:55:55 dist-test-slave-4mtw krb5kdc[23568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:55.701614 23581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:55.702019 23581 webserver.cc:460] Webserver started at http://127.0.0.1:36789/ using document root <none> and password file <none>
I20260620 22:55:55.702224 23581 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:55.702270 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:55.702386 23581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:55.703348 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "da4673e47a5c447198cb3719e1a3e45c"
format_stamp: "Formatted at 2026-06-20 22:55:55 on dist-test-slave-4mtw"
server_key: "c1f9779616408bcbacdb05c30eb23831"
server_key_iv: "f5abed87d3a75aa9158f788a10ba31d1"
server_key_version: "encryptionkey@0"
I20260620 22:55:55.703670 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "da4673e47a5c447198cb3719e1a3e45c"
format_stamp: "Formatted at 2026-06-20 22:55:55 on dist-test-slave-4mtw"
server_key: "c1f9779616408bcbacdb05c30eb23831"
server_key_iv: "f5abed87d3a75aa9158f788a10ba31d1"
server_key_version: "encryptionkey@0"
I20260620 22:55:55.704842 23581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:55:55.705458 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:55.705610 23581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:55:55.705703 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "da4673e47a5c447198cb3719e1a3e45c"
format_stamp: "Formatted at 2026-06-20 22:55:55 on dist-test-slave-4mtw"
server_key: "c1f9779616408bcbacdb05c30eb23831"
server_key_iv: "f5abed87d3a75aa9158f788a10ba31d1"
server_key_version: "encryptionkey@0"
I20260620 22:55:55.705770 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:55.729239 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:55.730923 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:55.731060 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:55.734727 23581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33231
I20260620 22:55:55.734759 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33231 every 8 connection(s)
I20260620 22:55:55.735080 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:55.735975 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:55.736174 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23581
I20260620 22:55:55.736289 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:55.736502 2410 external_mini_cluster.cc:1428] Setting key ebd35dbc3c6aa1e186f12fe92498121b
I20260620 22:55:55.739147 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c: Bootstrap starting.
I20260620 22:55:55.739761 23649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:55.740062 23649 log.cc:824] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:55.740731 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c: No bootstrap required, opened a new log
Jun 20 22:55:55 dist-test-slave-4mtw krb5kdc[23568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 22:55:55.742967 23649 raft_consensus.cc:348] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4673e47a5c447198cb3719e1a3e45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260620 22:55:55.743120 23649 raft_consensus.cc:374] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:55.743146 23649 raft_consensus.cc:729] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4673e47a5c447198cb3719e1a3e45c, State: Initialized, Role: FOLLOWER
I20260620 22:55:55.743268 23649 consensus_queue.cc:260] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4673e47a5c447198cb3719e1a3e45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260620 22:55:55.743324 23649 raft_consensus.cc:388] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:55.743346 23649 raft_consensus.cc:482] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:55.743391 23649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:55.744032 23649 raft_consensus.cc:504] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4673e47a5c447198cb3719e1a3e45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260620 22:55:55.744146 23649 leader_election.cc:302] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da4673e47a5c447198cb3719e1a3e45c; no voters:
I20260620 22:55:55.744315 23649 leader_election.cc:288] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:55.744374 23654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:55.744521 23654 raft_consensus.cc:686] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [term 1 LEADER]: Becoming Leader. State: Replica: da4673e47a5c447198cb3719e1a3e45c, State: Running, Role: LEADER
I20260620 22:55:55.744654 23649 sys_catalog.cc:564] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:55.744673 23654 consensus_queue.cc:237] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4673e47a5c447198cb3719e1a3e45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } }
I20260620 22:55:55.745186 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da4673e47a5c447198cb3719e1a3e45c. Latest consensus state: current_term: 1 leader_uuid: "da4673e47a5c447198cb3719e1a3e45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4673e47a5c447198cb3719e1a3e45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } } }
I20260620 22:55:55.745263 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:55.745750 23661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:55.745824 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da4673e47a5c447198cb3719e1a3e45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4673e47a5c447198cb3719e1a3e45c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33231 } } }
I20260620 22:55:55.745939 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:55.746295 23661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:55.747231 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:55.737025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:51739 (local address 127.0.0.1:33231)
0620 22:55:55.737185 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:55.737196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:55.737223 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:55:55.737663 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:55.737670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:55.737724 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:55.737837 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:55.738563 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:55.739497 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:55.740450 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:55.740930 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:55.741017 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:55.743518 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:55.743522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:55.743523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:55.743535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:55.745406 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:55.745883 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:55.745886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:55.745886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:55.745994 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:55.746242 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:55.746243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:55.746243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:55.746564 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:55.746713 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:55.746953 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 22:55:55.747050 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260620 22:55:55.747829 23661 catalog_manager.cc:1347] Generated new cluster ID: 10b7124e350640c485c5c6d82a1a900e
I20260620 22:55:55.747877 23661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:55.748281 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:55.756551 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:55.749606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:37271 (local address 127.0.0.1:33231)
0620 22:55:55.749638 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:55.749640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:55.749740 (+ 100us) server_negotiation.cc:407] Connection header received
0620 22:55:55.749790 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:55.749791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:55.749800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:55.749853 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:55.750481 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:55.751200 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:55.752199 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:55.752521 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:55.752574 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:55.753930 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:55.753931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:55.753932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:55.753938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:55.755399 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:55.755798 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:55.755798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:55.755799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:55.755853 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:55.756072 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:55.756073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:55.756073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:55.756182 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:55.756276 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:55.756407 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 22:55:55.756474 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:55:55.757158 23661 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:55.757632 23661 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:55.761530 23661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da4673e47a5c447198cb3719e1a3e45c: Generated new TSK 0
I20260620 22:55:55.761710 23661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:55.766314 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23581
W20260620 22:55:55.770252 23566 connection.cc:535] client connection to 127.0.0.1:33231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33231): Connection reset by peer (error 104)
2026-06-20T22:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:55 dist-test-slave-4mtw krb5kdc[23680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:55 dist-test-slave-4mtw krb5kdc[23680](info): set up 2 sockets
Jun 20 22:55:55 dist-test-slave-4mtw krb5kdc[23680](info): commencing operation
krb5kdc: starting...
W20260620 22:55:56.704363 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:56 dist-test-slave-4mtw krb5kdc[23680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:56.811879 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:56.888935 23693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:56.889140 23693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:56.889170 23693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:56.890285 23693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:56.890326 23693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:56.890347 23693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:56.890362 23693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:56.890378 23693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:56.891870 23693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:56.892108 23693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:56.892319 23693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:56.896812 23698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:56.896835 23699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:56.896926 23693 server_base.cc:1034] running on GCE node
W20260620 22:55:56.896836 23701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:56.897197 23693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:56.897482 23693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:56.898627 23693 hybrid_clock.cc:648] HybridClock initialized: now 1781996156898612 us; error 27 us; skew 500 ppm
Jun 20 22:55:56 dist-test-slave-4mtw krb5kdc[23680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:56.900933 23693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:56.901362 23693 webserver.cc:460] Webserver started at http://127.0.0.1:44839/ using document root <none> and password file <none>
I20260620 22:55:56.901559 23693 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:56.901607 23693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:56.901731 23693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:56.902736 23693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "6b494bfe4a8d475a899f6ea7994ad9a2"
format_stamp: "Formatted at 2026-06-20 22:55:56 on dist-test-slave-4mtw"
server_key: "00357e684214052d8c1f51abae01a4eb"
server_key_iv: "f700e15b7d7aefffb5b4214308465627"
server_key_version: "encryptionkey@0"
I20260620 22:55:56.903056 23693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "6b494bfe4a8d475a899f6ea7994ad9a2"
format_stamp: "Formatted at 2026-06-20 22:55:56 on dist-test-slave-4mtw"
server_key: "00357e684214052d8c1f51abae01a4eb"
server_key_iv: "f700e15b7d7aefffb5b4214308465627"
server_key_version: "encryptionkey@0"
I20260620 22:55:56.904227 23693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:56.904942 23708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:56.905086 23693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:56.905169 23693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "6b494bfe4a8d475a899f6ea7994ad9a2"
format_stamp: "Formatted at 2026-06-20 22:55:56 on dist-test-slave-4mtw"
server_key: "00357e684214052d8c1f51abae01a4eb"
server_key_iv: "f700e15b7d7aefffb5b4214308465627"
server_key_version: "encryptionkey@0"
I20260620 22:55:56.905226 23693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:56.930593 23693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:56.932775 23693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:56.932909 23693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:56.936480 23693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41001
I20260620 22:55:56.936510 23760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41001 every 8 connection(s)
I20260620 22:55:56.936854 23693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:56.937639 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:56.939769 23761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2: Bootstrap starting.
I20260620 22:55:56.940269 23761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:56.940493 23761 log.cc:824] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:56.941104 23761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2: No bootstrap required, opened a new log
I20260620 22:55:56.942580 23761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41001 } }
I20260620 22:55:56.942709 23761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:56.942729 23761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b494bfe4a8d475a899f6ea7994ad9a2, State: Initialized, Role: FOLLOWER
I20260620 22:55:56.942834 23761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41001 } }
I20260620 22:55:56.942906 23761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:56.942934 23761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:56.942965 23761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:56.943394 23761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41001 } }
I20260620 22:55:56.943480 23761 leader_election.cc:302] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b494bfe4a8d475a899f6ea7994ad9a2; no voters:
I20260620 22:55:56.943598 23761 leader_election.cc:288] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:56.943645 23764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:56.943794 23761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:56.943817 23764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 6b494bfe4a8d475a899f6ea7994ad9a2, State: Running, Role: LEADER
I20260620 22:55:56.943986 23764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41001 } }
I20260620 22:55:56.944456 23766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b494bfe4a8d475a899f6ea7994ad9a2. Latest consensus state: current_term: 1 leader_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41001 } } }
I20260620 22:55:56.944550 23766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:56.944782 23765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b494bfe4a8d475a899f6ea7994ad9a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41001 } } }
I20260620 22:55:56.945017 23776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:56.945017 23765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:56.945513 23776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:56.945854 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23693
I20260620 22:55:56.945964 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:56.946138 2410 external_mini_cluster.cc:1428] Setting key 2a1f5442683e2f07a6357b81842b8ec1
I20260620 22:55:56.947194 23776 catalog_manager.cc:1347] Generated new cluster ID: e22844139b35454e985326314538dfa8
I20260620 22:55:56.947235 23776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:56.953828 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:56.946562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:56415 (local address 127.0.0.1:41001)
0620 22:55:56.948832 (+ 2270us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:56.948841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:56.948878 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:55:56.949001 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:56.949008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:56.949065 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:56.949216 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:56.949375 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:56.950780 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:56.951796 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:56.952433 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:56.952544 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:56.953186 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:56.953190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:56.953191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:56.953391 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:56.953434 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:56.953449 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:56.953534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:56.953599 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 22:55:56.953627 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2213,"thread_start_us":108,"threads_started":1}
I20260620 22:55:56.954736 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:56.968197 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:56.956118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:56719 (local address 127.0.0.1:41001)
0620 22:55:56.956147 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:56.956149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:56.964871 (+ 8722us) server_negotiation.cc:407] Connection header received
0620 22:55:56.964939 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:56.964940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:56.964950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:56.965006 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:56.965169 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:56.965903 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:56.966923 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:56.967234 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:56.967285 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:56.967949 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:56.967950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:56.967950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:56.967955 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:56.967969 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:56.967973 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:56.968020 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:56.968104 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 22:55:56.968117 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 22:55:56.969982 23776 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:56.970453 23776 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:56.974359 23776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b494bfe4a8d475a899f6ea7994ad9a2: Generated new TSK 0
I20260620 22:55:56.974535 23776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:56.977350 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23693
W20260620 22:55:56.981045 23678 connection.cc:535] client connection to 127.0.0.1:41001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41001): Connection reset by peer (error 104)
2026-06-20T22:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:57 dist-test-slave-4mtw krb5kdc[23792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:57 dist-test-slave-4mtw krb5kdc[23792](info): set up 2 sockets
Jun 20 22:55:57 dist-test-slave-4mtw krb5kdc[23792](info): commencing operation
krb5kdc: starting...
W20260620 22:55:57.912534 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:57 dist-test-slave-4mtw krb5kdc[23792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 22:55:58.017201 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:58.093750 23805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:58.093947 23805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:58.093983 23805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:58.095093 23805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:58.095131 23805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:58.095150 23805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:58.095167 23805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:58.095183 23805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:58.096597 23805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:58.096848 23805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:58.097044 23805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:55:58.101332 23810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:58.101415 23805 server_base.cc:1034] running on GCE node
W20260620 22:55:58.101332 23813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:58.101372 23811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:58.101796 23805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:58.102058 23805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:58.103188 23805 hybrid_clock.cc:648] HybridClock initialized: now 1781996158103174 us; error 25 us; skew 500 ppm
Jun 20 22:55:58 dist-test-slave-4mtw krb5kdc[23792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:58.105415 23805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 22:55:58.105815 23805 webserver.cc:460] Webserver started at http://127.0.0.1:45889/ using document root <none> and password file <none>
I20260620 22:55:58.106011 23805 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:58.106056 23805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:58.106176 23805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:58.107136 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "dd69a4e13b244287bbc8490b22c7a7c0"
format_stamp: "Formatted at 2026-06-20 22:55:58 on dist-test-slave-4mtw"
server_key: "4f2b92139abacc35acf2a5532391cb99"
server_key_iv: "9a8ea89e60c37e15affb7740ea629256"
server_key_version: "encryptionkey@0"
I20260620 22:55:58.107441 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "dd69a4e13b244287bbc8490b22c7a7c0"
format_stamp: "Formatted at 2026-06-20 22:55:58 on dist-test-slave-4mtw"
server_key: "4f2b92139abacc35acf2a5532391cb99"
server_key_iv: "9a8ea89e60c37e15affb7740ea629256"
server_key_version: "encryptionkey@0"
I20260620 22:55:58.108587 23805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:58.109232 23820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:58.109370 23805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:58.109447 23805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "dd69a4e13b244287bbc8490b22c7a7c0"
format_stamp: "Formatted at 2026-06-20 22:55:58 on dist-test-slave-4mtw"
server_key: "4f2b92139abacc35acf2a5532391cb99"
server_key_iv: "9a8ea89e60c37e15affb7740ea629256"
server_key_version: "encryptionkey@0"
I20260620 22:55:58.109512 23805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:58.112562 23805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:58.114135 23805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:58.114254 23805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:58.117812 23805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39751
I20260620 22:55:58.117851 23872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39751 every 8 connection(s)
I20260620 22:55:58.118114 23805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:58.118948 23873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:58.120852 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23805
I20260620 22:55:58.120976 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:58.121048 23873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0: Bootstrap starting.
I20260620 22:55:58.121157 2410 external_mini_cluster.cc:1428] Setting key 6501b839b090e61f86d88f7909bbe1b3
I20260620 22:55:58.121660 23873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:58.121918 23873 log.cc:824] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:58.122586 23873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0: No bootstrap required, opened a new log
I20260620 22:55:58.123378 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:58.121583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:37085 (local address 127.0.0.1:39751)
0620 22:55:58.121754 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:58.121764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:58.121793 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:55:58.122267 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:58.122277 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:58.122311 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:58.122435 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:58.123000 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:58.123003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:58.123003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:58.123007 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:58.123040 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:58.123050 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:58.123110 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:58.123167 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 22:55:58.123199 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260620 22:55:58.124174 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:58.124712 23873 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39751 } }
I20260620 22:55:58.124871 23873 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:58.124913 23873 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd69a4e13b244287bbc8490b22c7a7c0, State: Initialized, Role: FOLLOWER
I20260620 22:55:58.125023 23873 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39751 } }
I20260620 22:55:58.125104 23873 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:58.125138 23873 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:58.125182 23873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:58.125804 23873 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39751 } }
I20260620 22:55:58.125897 23873 leader_election.cc:302] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd69a4e13b244287bbc8490b22c7a7c0; no voters:
I20260620 22:55:58.126035 23873 leader_election.cc:288] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:58.126070 23883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:58.126196 23883 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [term 1 LEADER]: Becoming Leader. State: Replica: dd69a4e13b244287bbc8490b22c7a7c0, State: Running, Role: LEADER
I20260620 22:55:58.126292 23873 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:58.126317 23883 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39751 } }
I20260620 22:55:58.126642 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:58.125640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:39057 (local address 127.0.0.1:39751)
0620 22:55:58.125675 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:58.125676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:58.125794 (+ 118us) server_negotiation.cc:407] Connection header received
0620 22:55:58.125846 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:58.125846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:58.125851 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:58.125901 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:58.126444 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:58.126445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:58.126445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:55:58.126449 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:58.126461 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:55:58.126464 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:58.126501 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:58.126574 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:55:58.126585 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:55:58.126708 23884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39751 } } }
I20260620 22:55:58.126785 23884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:58.127170 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd69a4e13b244287bbc8490b22c7a7c0. Latest consensus state: current_term: 1 leader_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd69a4e13b244287bbc8490b22c7a7c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39751 } } }
I20260620 22:55:58.127238 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0 [sys.catalog]: This master's current role is: LEADER
W20260620 22:55:58.127823 23899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:55:58.127864 23899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:55:58.127944 23891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:58.128116 23891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:55:58.129436 23891 catalog_manager.cc:1347] Generated new cluster ID: 79c0a62a34904ac5a63f60ce7e32477b
I20260620 22:55:58.129531 23891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:58.146828 23891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:58.146989 23891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:58.153008 23891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd69a4e13b244287bbc8490b22c7a7c0: Generated new TSK 0
I20260620 22:55:58.153172 23891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:58.162626 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23805
2026-06-20T22:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:58 dist-test-slave-4mtw krb5kdc[23904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:58 dist-test-slave-4mtw krb5kdc[23904](info): set up 2 sockets
Jun 20 22:55:58 dist-test-slave-4mtw krb5kdc[23904](info): commencing operation
krb5kdc: starting...
W20260620 22:55:59.096371 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:55:59 dist-test-slave-4mtw krb5kdc[23904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:55:59.204227 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:43821
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:34433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:55:59.280869 23917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:55:59.281082 23917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:55:59.281114 23917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:55:59.282255 23917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:55:59.282298 23917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:55:59.282317 23917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:55:59.282333 23917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:55:59.282347 23917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:55:59.283914 23917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:34433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:43821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:55:59.284168 23917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:55:59.284369 23917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:55:59.288854 23917 server_base.cc:1034] running on GCE node
W20260620 22:55:59.288908 23923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:59.288823 23925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:55:59.288822 23922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:55:59.289233 23917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:55:59.289510 23917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:55:59.290647 23917 hybrid_clock.cc:648] HybridClock initialized: now 1781996159290633 us; error 29 us; skew 500 ppm
Jun 20 22:55:59 dist-test-slave-4mtw krb5kdc[23904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:55:59.292930 23917 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:55:59.293313 23917 webserver.cc:460] Webserver started at http://127.2.90.190:38269/ using document root <none> and password file <none>
I20260620 22:55:59.293498 23917 fs_manager.cc:359] Metadata directory not provided
I20260620 22:55:59.293540 23917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:55:59.293656 23917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:55:59.294628 23917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "2b939de739b3472eb2848a4fd183ab35"
format_stamp: "Formatted at 2026-06-20 22:55:59 on dist-test-slave-4mtw"
server_key: "93938818ffed937398eae61440070730"
server_key_iv: "b0c554bc53d6356c333e6aeffea7d818"
server_key_version: "encryptionkey@0"
I20260620 22:55:59.294934 23917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "2b939de739b3472eb2848a4fd183ab35"
format_stamp: "Formatted at 2026-06-20 22:55:59 on dist-test-slave-4mtw"
server_key: "93938818ffed937398eae61440070730"
server_key_iv: "b0c554bc53d6356c333e6aeffea7d818"
server_key_version: "encryptionkey@0"
I20260620 22:55:59.296082 23917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:55:59.296690 23932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:59.296831 23917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 22:55:59.296911 23917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "2b939de739b3472eb2848a4fd183ab35"
format_stamp: "Formatted at 2026-06-20 22:55:59 on dist-test-slave-4mtw"
server_key: "93938818ffed937398eae61440070730"
server_key_iv: "b0c554bc53d6356c333e6aeffea7d818"
server_key_version: "encryptionkey@0"
I20260620 22:55:59.296978 23917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:55:59.305886 23917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:55:59.307758 23917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:55:59.307883 23917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:55:59.311444 23917 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:43821
I20260620 22:55:59.311481 23984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:43821 every 8 connection(s)
I20260620 22:55:59.311767 23917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:55:59.312507 23985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:55:59.314633 23985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35: Bootstrap starting.
I20260620 22:55:59.315121 23985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35: Neither blocks nor log segments found. Creating new log.
I20260620 22:55:59.315333 23985 log.cc:824] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35: Log is configured to *not* fsync() on all Append() calls
I20260620 22:55:59.315888 23985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35: No bootstrap required, opened a new log
I20260620 22:55:59.317417 23985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b939de739b3472eb2848a4fd183ab35" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 43821 } }
I20260620 22:55:59.317564 23985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:55:59.317585 23985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b939de739b3472eb2848a4fd183ab35, State: Initialized, Role: FOLLOWER
I20260620 22:55:59.317734 23985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b939de739b3472eb2848a4fd183ab35" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 43821 } }
I20260620 22:55:59.317788 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 23917
I20260620 22:55:59.317809 23985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:55:59.317842 23985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:55:59.317890 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:55:59.317889 23985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:55:59.318055 2410 external_mini_cluster.cc:1428] Setting key b9b9a232d5c7b959b2c0cc3e6a2d2d1a
I20260620 22:55:59.318842 23985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b939de739b3472eb2848a4fd183ab35" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 43821 } }
I20260620 22:55:59.318956 23985 leader_election.cc:302] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b939de739b3472eb2848a4fd183ab35; no voters:
I20260620 22:55:59.319106 23985 leader_election.cc:288] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:55:59.319129 23990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:55:59.319221 23990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [term 1 LEADER]: Becoming Leader. State: Replica: 2b939de739b3472eb2848a4fd183ab35, State: Running, Role: LEADER
I20260620 22:55:59.319339 23985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:55:59.319350 23990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b939de739b3472eb2848a4fd183ab35" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 43821 } }
I20260620 22:55:59.319722 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b939de739b3472eb2848a4fd183ab35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b939de739b3472eb2848a4fd183ab35" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 43821 } } }
I20260620 22:55:59.319738 23991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b939de739b3472eb2848a4fd183ab35. Latest consensus state: current_term: 1 leader_uuid: "2b939de739b3472eb2848a4fd183ab35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b939de739b3472eb2848a4fd183ab35" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 43821 } } }
I20260620 22:55:59.319957 23991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:59.319871 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35 [sys.catalog]: This master's current role is: LEADER
I20260620 22:55:59.320206 23999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:55:59.320709 23999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 22:55:59 dist-test-slave-4mtw krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:55:59.326033 23999 catalog_manager.cc:1347] Generated new cluster ID: fa50d0b0984b4160a7e10926696ca740
I20260620 22:55:59.326087 23999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:55:59.327931 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:59.318455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46237 (local address 127.2.90.190:43821)
0620 22:55:59.318634 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:59.318644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:59.318675 (+ 31us) server_negotiation.cc:407] Connection header received
0620 22:55:59.318783 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:59.318790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:59.318843 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:59.318985 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:59.319679 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:59.321056 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:59.321854 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:59.322417 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:59.322535 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:59.324801 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:59.324806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:59.324806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:59.324816 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:59.326539 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:59.326881 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:59.326884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:59.326884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:59.326976 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:59.327172 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:59.327173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:59.327174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:59.327389 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:59.327553 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:59.327655 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 22:55:59.327751 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260620 22:55:59.329329 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:55:59.335590 23999 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:55:59.336232 23999 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:55:59.338406 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:55:59.332990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34733 (local address 127.2.90.190:43821)
0620 22:55:59.333033 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:55:59.333035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:55:59.333152 (+ 117us) server_negotiation.cc:407] Connection header received
0620 22:55:59.333204 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:55:59.333204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:55:59.333214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:55:59.333257 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:55:59.333720 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:59.334188 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:59.334841 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:55:59.335054 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:55:59.335106 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:55:59.336267 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:55:59.336268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:55:59.336268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:55:59.336275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:55:59.337416 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:59.337738 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:59.337739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:59.337739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:59.337787 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:55:59.337972 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:55:59.337973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:55:59.337974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:55:59.338068 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:55:59.338164 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 22:55:59.338271 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 22:55:59.338325 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260620 22:55:59.344913 23999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b939de739b3472eb2848a4fd183ab35: Generated new TSK 0
I20260620 22:55:59.345093 23999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:55:59.348009 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 23917
W20260620 22:55:59.351578 23902 connection.cc:535] client connection to 127.2.90.190:43821 recv error: Network error: failed to read from TLS socket (remote: 127.2.90.190:43821): Connection reset by peer (error 104)
2026-06-20T22:55:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:55:59 dist-test-slave-4mtw krb5kdc[24016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:55:59 dist-test-slave-4mtw krb5kdc[24016](info): set up 2 sockets
Jun 20 22:55:59 dist-test-slave-4mtw krb5kdc[24016](info): commencing operation
krb5kdc: starting...
W20260620 22:56:00.288507 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:00 dist-test-slave-4mtw krb5kdc[24016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:00.393363 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39247
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:36271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:00.470103 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:00.470294 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:00.470315 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:00.471396 24029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:00.471429 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:00.471442 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:00.471453 24029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:00.471467 24029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:00.472910 24029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:36271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:39247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:00.473100 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:00.473261 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:00.477661 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:00.477655 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:00.477675 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:00.477756 24029 server_base.cc:1034] running on GCE node
I20260620 22:56:00.478147 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:00.478407 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:00.479547 24029 hybrid_clock.cc:648] HybridClock initialized: now 1781996160479534 us; error 32 us; skew 500 ppm
Jun 20 22:56:00 dist-test-slave-4mtw krb5kdc[24016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:00.484508 24029 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:00.484936 24029 webserver.cc:460] Webserver started at http://127.2.90.190:40811/ using document root <none> and password file <none>
I20260620 22:56:00.485154 24029 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:00.485213 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:00.485333 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:00.486311 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "1ee80b0ef458499ba62906b29fbac289"
format_stamp: "Formatted at 2026-06-20 22:56:00 on dist-test-slave-4mtw"
server_key: "619472c309d9bc30467bce8e94c2b050"
server_key_iv: "0b20b60503d498d84713ea33cbb6b5c8"
server_key_version: "encryptionkey@0"
I20260620 22:56:00.486656 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "1ee80b0ef458499ba62906b29fbac289"
format_stamp: "Formatted at 2026-06-20 22:56:00 on dist-test-slave-4mtw"
server_key: "619472c309d9bc30467bce8e94c2b050"
server_key_iv: "0b20b60503d498d84713ea33cbb6b5c8"
server_key_version: "encryptionkey@0"
I20260620 22:56:00.487811 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:00.488415 24044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:00.488565 24029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:00.488646 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "1ee80b0ef458499ba62906b29fbac289"
format_stamp: "Formatted at 2026-06-20 22:56:00 on dist-test-slave-4mtw"
server_key: "619472c309d9bc30467bce8e94c2b050"
server_key_iv: "0b20b60503d498d84713ea33cbb6b5c8"
server_key_version: "encryptionkey@0"
I20260620 22:56:00.488731 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:00.496475 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:00.498023 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:00.498143 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:00.501610 24029 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:39247
I20260620 22:56:00.501643 24096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:39247 every 8 connection(s)
I20260620 22:56:00.501925 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:00.502658 24097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:00.504786 24097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289: Bootstrap starting.
I20260620 22:56:00.505301 24097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:00.505539 24097 log.cc:824] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:00.506131 24097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289: No bootstrap required, opened a new log
I20260620 22:56:00.506919 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24029
I20260620 22:56:00.507046 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:00.507208 2410 external_mini_cluster.cc:1428] Setting key 4bbe58e923f3961a6c51e4a4bee89a7a
I20260620 22:56:00.507917 24097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee80b0ef458499ba62906b29fbac289" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39247 } }
I20260620 22:56:00.508091 24097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:00.508134 24097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee80b0ef458499ba62906b29fbac289, State: Initialized, Role: FOLLOWER
I20260620 22:56:00.508313 24097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee80b0ef458499ba62906b29fbac289" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39247 } }
I20260620 22:56:00.508399 24097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:00.508456 24097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:00.508512 24097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:00.509193 24097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee80b0ef458499ba62906b29fbac289" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39247 } }
I20260620 22:56:00.509294 24097 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee80b0ef458499ba62906b29fbac289; no voters:
I20260620 22:56:00.509442 24097 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:00.509481 24102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:00.509615 24102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee80b0ef458499ba62906b29fbac289, State: Running, Role: LEADER
I20260620 22:56:00.509665 24097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:00.509739 24102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee80b0ef458499ba62906b29fbac289" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39247 } }
I20260620 22:56:00.510049 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee80b0ef458499ba62906b29fbac289. Latest consensus state: current_term: 1 leader_uuid: "1ee80b0ef458499ba62906b29fbac289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee80b0ef458499ba62906b29fbac289" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39247 } } }
I20260620 22:56:00.510253 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:00.510183 24104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee80b0ef458499ba62906b29fbac289" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee80b0ef458499ba62906b29fbac289" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 39247 } } }
I20260620 22:56:00.510612 24104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:00.510771 24111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:00.511327 24111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:00.512844 24111 catalog_manager.cc:1347] Generated new cluster ID: 423a1ab9a6f04b7099df286cab532c83
I20260620 22:56:00.512897 24111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:00.513417 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:00.507653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46327 (local address 127.2.90.190:39247)
0620 22:56:00.507836 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:00.507845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:00.507877 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:56:00.507988 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:00.507994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:00.508047 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:00.508187 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:00.508388 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:00.510006 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:00.511131 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:00.511705 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:00.511803 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:00.512644 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:00.512648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:00.512648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:00.512914 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:00.512951 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:00.512965 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:00.513040 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:00.513203 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 22:56:00.513228 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260620 22:56:00.517582 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:00.520406 24111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:00.521116 24111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:00.521958 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:00.519017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36567 (local address 127.2.90.190:39247)
0620 22:56:00.519053 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:00.519055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:00.519153 (+ 98us) server_negotiation.cc:407] Connection header received
0620 22:56:00.519200 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:00.519201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:00.519211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:00.519265 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:00.519398 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:00.520051 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:00.520923 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:00.521119 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:00.521169 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:00.521741 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:00.521742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:00.521742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:00.521746 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:00.521760 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:00.521764 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:00.521811 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:00.521877 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:56:00.521887 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:00.527678 24111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee80b0ef458499ba62906b29fbac289: Generated new TSK 0
I20260620 22:56:00.527829 24111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:00.532469 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24029
W20260620 22:56:00.536058 24014 connection.cc:535] client connection to 127.2.90.190:39247 recv error: Network error: failed to read from TLS socket (remote: 127.2.90.190:39247): Connection reset by peer (error 104)
2026-06-20T22:56:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:00 dist-test-slave-4mtw krb5kdc[24128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:00 dist-test-slave-4mtw krb5kdc[24128](info): set up 2 sockets
Jun 20 22:56:00 dist-test-slave-4mtw krb5kdc[24128](info): commencing operation
krb5kdc: starting...
W20260620 22:56:01.472348 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:01 dist-test-slave-4mtw krb5kdc[24128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:01.579039 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:34975
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:01.656055 24141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:01.656245 24141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:01.656266 24141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:01.657387 24141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:01.657426 24141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:01.657446 24141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:01.657466 24141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:01.657490 24141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:01.658903 24141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:34975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:01.659113 24141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:01.659308 24141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 22:56:01.663713 24141 server_base.cc:1034] running on GCE node
W20260620 22:56:01.663682 24146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:01.663662 24149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:01.663734 24147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:01.664141 24141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:01.664383 24141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:01.665512 24141 hybrid_clock.cc:648] HybridClock initialized: now 1781996161665495 us; error 34 us; skew 500 ppm
Jun 20 22:56:01 dist-test-slave-4mtw krb5kdc[24128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996161, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:01.667732 24141 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:01.668108 24141 webserver.cc:460] Webserver started at http://127.2.90.190:35755/ using document root <none> and password file <none>
I20260620 22:56:01.668299 24141 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:01.668339 24141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:01.668428 24141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:01.669432 24141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "82375f2a4eec4952a2eef8316c7302d1"
format_stamp: "Formatted at 2026-06-20 22:56:01 on dist-test-slave-4mtw"
server_key: "e1640c40fe4a20aa96e763b742147c1a"
server_key_iv: "f55229b991ad40ef49e0d5cea86df8e3"
server_key_version: "encryptionkey@0"
I20260620 22:56:01.669706 24141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "82375f2a4eec4952a2eef8316c7302d1"
format_stamp: "Formatted at 2026-06-20 22:56:01 on dist-test-slave-4mtw"
server_key: "e1640c40fe4a20aa96e763b742147c1a"
server_key_iv: "f55229b991ad40ef49e0d5cea86df8e3"
server_key_version: "encryptionkey@0"
I20260620 22:56:01.670796 24141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:56:01.671375 24156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:01.671514 24141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:56:01.671591 24141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "82375f2a4eec4952a2eef8316c7302d1"
format_stamp: "Formatted at 2026-06-20 22:56:01 on dist-test-slave-4mtw"
server_key: "e1640c40fe4a20aa96e763b742147c1a"
server_key_iv: "f55229b991ad40ef49e0d5cea86df8e3"
server_key_version: "encryptionkey@0"
I20260620 22:56:01.671649 24141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:01.674832 24141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:01.676538 24141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:01.676658 24141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:01.680298 24141 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:34975
I20260620 22:56:01.680332 24208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:34975 every 8 connection(s)
I20260620 22:56:01.680645 24141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:01.681444 24209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:01.682960 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24141
I20260620 22:56:01.683061 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:01.683219 2410 external_mini_cluster.cc:1428] Setting key cb4e266ad4600a80bccd499d683e5630
I20260620 22:56:01.683640 24209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1: Bootstrap starting.
I20260620 22:56:01.684193 24209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:01.684464 24209 log.cc:824] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:01.685179 24209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1: No bootstrap required, opened a new log
I20260620 22:56:01.685460 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:01.683661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46147 (local address 127.2.90.190:34975)
0620 22:56:01.683810 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:01.683819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:01.683844 (+ 25us) server_negotiation.cc:407] Connection header received
0620 22:56:01.684260 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:01.684268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:01.684301 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:01.684414 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:01.685071 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:01.685075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:01.685075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:01.685079 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:01.685122 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:01.685135 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:01.685225 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:01.685254 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 22:56:01.685292 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260620 22:56:01.686295 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:01.687228 24209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82375f2a4eec4952a2eef8316c7302d1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34975 } }
I20260620 22:56:01.687410 24209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:01.687470 24209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82375f2a4eec4952a2eef8316c7302d1, State: Initialized, Role: FOLLOWER
I20260620 22:56:01.687587 24209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82375f2a4eec4952a2eef8316c7302d1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34975 } }
I20260620 22:56:01.687681 24209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:01.687719 24209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:01.687763 24209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:01.688359 24209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82375f2a4eec4952a2eef8316c7302d1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34975 } }
I20260620 22:56:01.688467 24209 leader_election.cc:302] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82375f2a4eec4952a2eef8316c7302d1; no voters:
I20260620 22:56:01.688511 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:01.687557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52477 (local address 127.2.90.190:34975)
0620 22:56:01.687590 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:01.687592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:01.687713 (+ 121us) server_negotiation.cc:407] Connection header received
0620 22:56:01.687777 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:01.687778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:01.687782 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:01.687838 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:01.688317 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:01.688319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:01.688319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:01.688322 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:01.688334 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:01.688338 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:01.688390 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:01.688425 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:56:01.688435 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:56:01.688619 24209 leader_election.cc:288] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:01.688637 24219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:01.688760 24219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [term 1 LEADER]: Becoming Leader. State: Replica: 82375f2a4eec4952a2eef8316c7302d1, State: Running, Role: LEADER
I20260620 22:56:01.688836 24219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82375f2a4eec4952a2eef8316c7302d1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34975 } }
I20260620 22:56:01.688862 24209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:01.690219 24221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82375f2a4eec4952a2eef8316c7302d1. Latest consensus state: current_term: 1 leader_uuid: "82375f2a4eec4952a2eef8316c7302d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82375f2a4eec4952a2eef8316c7302d1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34975 } } }
I20260620 22:56:01.690357 24220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82375f2a4eec4952a2eef8316c7302d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82375f2a4eec4952a2eef8316c7302d1" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34975 } } }
I20260620 22:56:01.690371 24221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:01.690412 24220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1 [sys.catalog]: This master's current role is: LEADER
W20260620 22:56:01.690717 24233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:01.690768 24233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:01.690814 24234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:01.690966 24234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:01.692157 24234 catalog_manager.cc:1347] Generated new cluster ID: 5daa45097c8b42d3862f7c3e02ab2903
I20260620 22:56:01.692209 24234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:01.699800 24234 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:01.699935 24234 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:01.706887 24234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82375f2a4eec4952a2eef8316c7302d1: Generated new TSK 0
I20260620 22:56:01.707047 24234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:01.710980 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24141
2026-06-20T22:56:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:01 dist-test-slave-4mtw krb5kdc[24240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:01 dist-test-slave-4mtw krb5kdc[24240](info): set up 2 sockets
Jun 20 22:56:01 dist-test-slave-4mtw krb5kdc[24240](info): commencing operation
krb5kdc: starting...
W20260620 22:56:02.656427 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:02 dist-test-slave-4mtw krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:02.761302 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:34195
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:34195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:02.838069 24253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:02.838268 24253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:02.838300 24253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:02.839396 24253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:02.839435 24253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:02.839454 24253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:02.839469 24253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:02.839485 24253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:02.840947 24253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:34195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:34195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:02.841169 24253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:02.841396 24253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:02.845757 24258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:02.845781 24259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:02.845799 24261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:02.845945 24253 server_base.cc:1034] running on GCE node
I20260620 22:56:02.846158 24253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:02.846432 24253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:02.847575 24253 hybrid_clock.cc:648] HybridClock initialized: now 1781996162847562 us; error 29 us; skew 500 ppm
Jun 20 22:56:02 dist-test-slave-4mtw krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:02.849835 24253 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:02.850210 24253 webserver.cc:460] Webserver started at http://127.2.90.190:43635/ using document root <none> and password file <none>
I20260620 22:56:02.850431 24253 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:02.850474 24253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:02.850586 24253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:02.851563 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "a79300f7523e4e1185985097af2b8c75"
format_stamp: "Formatted at 2026-06-20 22:56:02 on dist-test-slave-4mtw"
server_key: "c135529547cf4aad78401a656cb2ba86"
server_key_iv: "2e18cacb6fd798e347236f4b93ae655e"
server_key_version: "encryptionkey@0"
I20260620 22:56:02.851862 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "a79300f7523e4e1185985097af2b8c75"
format_stamp: "Formatted at 2026-06-20 22:56:02 on dist-test-slave-4mtw"
server_key: "c135529547cf4aad78401a656cb2ba86"
server_key_iv: "2e18cacb6fd798e347236f4b93ae655e"
server_key_version: "encryptionkey@0"
I20260620 22:56:02.853072 24253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:02.853713 24268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:02.853868 24253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:56:02.853938 24253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "a79300f7523e4e1185985097af2b8c75"
format_stamp: "Formatted at 2026-06-20 22:56:02 on dist-test-slave-4mtw"
server_key: "c135529547cf4aad78401a656cb2ba86"
server_key_iv: "2e18cacb6fd798e347236f4b93ae655e"
server_key_version: "encryptionkey@0"
I20260620 22:56:02.854001 24253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:02.869277 24253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:02.871224 24253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:02.871356 24253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:02.874926 24253 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:34195
I20260620 22:56:02.874962 24320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:34195 every 8 connection(s)
I20260620 22:56:02.875255 24253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:02.876022 24321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:02.878120 24321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75: Bootstrap starting.
I20260620 22:56:02.878608 24321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:02.878819 24321 log.cc:824] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:02.879400 24321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75: No bootstrap required, opened a new log
I20260620 22:56:02.880991 24321 raft_consensus.cc:348] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79300f7523e4e1185985097af2b8c75" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34195 } }
I20260620 22:56:02.881119 24321 raft_consensus.cc:374] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:02.881138 24321 raft_consensus.cc:729] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79300f7523e4e1185985097af2b8c75, State: Initialized, Role: FOLLOWER
I20260620 22:56:02.881237 24321 consensus_queue.cc:260] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79300f7523e4e1185985097af2b8c75" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34195 } }
I20260620 22:56:02.881285 24321 raft_consensus.cc:388] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:02.881304 24321 raft_consensus.cc:482] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:02.881332 24321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:02.882050 24321 raft_consensus.cc:504] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79300f7523e4e1185985097af2b8c75" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34195 } }
I20260620 22:56:02.882125 24321 leader_election.cc:302] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a79300f7523e4e1185985097af2b8c75; no voters:
I20260620 22:56:02.882234 24321 leader_election.cc:288] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:02.882274 24324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:02.882411 24324 raft_consensus.cc:686] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [term 1 LEADER]: Becoming Leader. State: Replica: a79300f7523e4e1185985097af2b8c75, State: Running, Role: LEADER
I20260620 22:56:02.882440 24321 sys_catalog.cc:564] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:02.882495 24324 consensus_queue.cc:237] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79300f7523e4e1185985097af2b8c75" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34195 } }
I20260620 22:56:02.882812 24325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a79300f7523e4e1185985097af2b8c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79300f7523e4e1185985097af2b8c75" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34195 } } }
I20260620 22:56:02.882839 24326 sys_catalog.cc:455] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a79300f7523e4e1185985097af2b8c75. Latest consensus state: current_term: 1 leader_uuid: "a79300f7523e4e1185985097af2b8c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a79300f7523e4e1185985097af2b8c75" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 34195 } } }
I20260620 22:56:02.882943 24326 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:02.882918 24325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:02.883129 24333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:02.883639 24333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:02.885020 24333 catalog_manager.cc:1347] Generated new cluster ID: 4f9e46342bf9492ba3103af53f7296d4
I20260620 22:56:02.885061 24333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:02.885282 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24253
I20260620 22:56:02.885393 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:02.885593 2410 external_mini_cluster.cc:1428] Setting key eb1f78bf6de56087526a304f469890ac
I20260620 22:56:02.890734 24333 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:02.891424 24333 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 22:56:02 dist-test-slave-4mtw krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:02.896366 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:02.886184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47563 (local address 127.2.90.190:34195)
0620 22:56:02.886329 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:02.886337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:02.886364 (+ 27us) server_negotiation.cc:407] Connection header received
0620 22:56:02.886478 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:02.886484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:02.886542 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:02.886663 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:02.887356 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:02.888714 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:02.889783 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:02.890405 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:02.890501 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:02.893138 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:02.893142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:02.893143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:02.893154 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:02.894927 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:02.895273 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:02.895276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:02.895276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:02.895377 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:02.895537 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:02.895538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:02.895538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:02.895787 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:02.895914 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:02.896007 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:56:02.896143 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260620 22:56:02.897324 24333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a79300f7523e4e1185985097af2b8c75: Generated new TSK 0
I20260620 22:56:02.897403 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:02.897467 24333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:02.904518 24344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:02.898823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36739 (local address 127.2.90.190:34195)
0620 22:56:02.898876 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:02.898878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:02.898887 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:02.898942 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:02.898943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:02.898952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:02.899012 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:02.899442 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:02.900201 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:02.900911 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:02.901244 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:02.901305 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:02.902294 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:02.902295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:02.902295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:02.902300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:02.903608 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:02.903890 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:02.903891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:02.903891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:02.903951 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:02.904091 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:02.904092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:02.904092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:02.904205 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:02.904288 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:02.904354 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:56:02.904426 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 22:56:02.906229 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24253
2026-06-20T22:56:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:02 dist-test-slave-4mtw krb5kdc[24352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:02 dist-test-slave-4mtw krb5kdc[24352](info): set up 2 sockets
Jun 20 22:56:02 dist-test-slave-4mtw krb5kdc[24352](info): commencing operation
krb5kdc: starting...
W20260620 22:56:03.844381 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:03 dist-test-slave-4mtw krb5kdc[24352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:03.947486 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45733
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:04.024277 24365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:04.024471 24365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:04.024492 24365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:04.025765 24365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:04.025813 24365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:04.025833 24365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:04.025848 24365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:04.025864 24365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:04.027638 24365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:45733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:04.027878 24365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:04.028087 24365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:04.032408 24371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:04.032437 24370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:04.032457 24373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:04.032614 24365 server_base.cc:1034] running on GCE node
I20260620 22:56:04.032842 24365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:04.033087 24365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:04.034227 24365 hybrid_clock.cc:648] HybridClock initialized: now 1781996164034212 us; error 32 us; skew 500 ppm
Jun 20 22:56:04 dist-test-slave-4mtw krb5kdc[24352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:04.036463 24365 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:04.036864 24365 webserver.cc:460] Webserver started at http://127.2.90.190:34077/ using document root <none> and password file <none>
I20260620 22:56:04.037030 24365 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:04.037068 24365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:04.037154 24365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:04.038081 24365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "dbedf10b142249d5804df9e83aafced7"
format_stamp: "Formatted at 2026-06-20 22:56:04 on dist-test-slave-4mtw"
server_key: "11f41d35129f457aab84864b34335f7c"
server_key_iv: "60d2c2431da9954b512c5a1450eaa549"
server_key_version: "encryptionkey@0"
I20260620 22:56:04.038354 24365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "dbedf10b142249d5804df9e83aafced7"
format_stamp: "Formatted at 2026-06-20 22:56:04 on dist-test-slave-4mtw"
server_key: "11f41d35129f457aab84864b34335f7c"
server_key_iv: "60d2c2431da9954b512c5a1450eaa549"
server_key_version: "encryptionkey@0"
I20260620 22:56:04.039429 24365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:04.039994 24380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:04.040126 24365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:04.040201 24365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "dbedf10b142249d5804df9e83aafced7"
format_stamp: "Formatted at 2026-06-20 22:56:04 on dist-test-slave-4mtw"
server_key: "11f41d35129f457aab84864b34335f7c"
server_key_iv: "60d2c2431da9954b512c5a1450eaa549"
server_key_version: "encryptionkey@0"
I20260620 22:56:04.040252 24365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:04.048004 24365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:04.049746 24365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:04.049841 24365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:04.053300 24365 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:45733
I20260620 22:56:04.053347 24432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:45733 every 8 connection(s)
I20260620 22:56:04.053625 24365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:04.054379 24433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:04.056885 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7: Bootstrap starting.
I20260620 22:56:04.057399 24433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:04.057636 24433 log.cc:824] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:04.058226 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7: No bootstrap required, opened a new log
I20260620 22:56:04.059701 24433 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbedf10b142249d5804df9e83aafced7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45733 } }
I20260620 22:56:04.059839 24433 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:04.059873 24433 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbedf10b142249d5804df9e83aafced7, State: Initialized, Role: FOLLOWER
I20260620 22:56:04.060001 24433 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbedf10b142249d5804df9e83aafced7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45733 } }
I20260620 22:56:04.060061 24433 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:04.060091 24433 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:04.060135 24433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:04.060938 24433 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbedf10b142249d5804df9e83aafced7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45733 } }
I20260620 22:56:04.061028 24433 leader_election.cc:302] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbedf10b142249d5804df9e83aafced7; no voters:
I20260620 22:56:04.061172 24433 leader_election.cc:288] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:04.061383 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24365
I20260620 22:56:04.061393 24433 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:04.061501 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:04.061674 2410 external_mini_cluster.cc:1428] Setting key 3bde371f38b56f5081aeac611e197556
I20260620 22:56:04.061200 24436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:04.061934 24436 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [term 1 LEADER]: Becoming Leader. State: Replica: dbedf10b142249d5804df9e83aafced7, State: Running, Role: LEADER
I20260620 22:56:04.062026 24436 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbedf10b142249d5804df9e83aafced7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45733 } }
I20260620 22:56:04.062506 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbedf10b142249d5804df9e83aafced7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbedf10b142249d5804df9e83aafced7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45733 } } }
I20260620 22:56:04.062589 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:04.062678 24441 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbedf10b142249d5804df9e83aafced7. Latest consensus state: current_term: 1 leader_uuid: "dbedf10b142249d5804df9e83aafced7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbedf10b142249d5804df9e83aafced7" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 45733 } } }
I20260620 22:56:04.062804 24441 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7 [sys.catalog]: This master's current role is: LEADER
W20260620 22:56:04.063097 24452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:04.063148 24452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:04.063210 24453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:04.063385 24453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:04.067448 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:04.062277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52101 (local address 127.2.90.190:45733)
0620 22:56:04.063226 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:04.063235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:04.063265 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:56:04.063377 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:04.063385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:04.063446 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:04.063574 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:04.063805 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:04.064751 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:04.065818 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:04.066215 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:04.066285 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:04.066869 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:04.066872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:04.066872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:04.067021 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:04.067050 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:04.067061 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:04.067107 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:04.067188 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 22:56:04.067234 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":68,"threads_started":1}
I20260620 22:56:04.067987 24453 catalog_manager.cc:1347] Generated new cluster ID: cc95ce06c7d2422391f07ccd2cccd380
I20260620 22:56:04.068030 24453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:04.068320 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:04.073033 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:04.069880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58445 (local address 127.2.90.190:45733)
0620 22:56:04.069915 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:04.069917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:04.070006 (+ 89us) server_negotiation.cc:407] Connection header received
0620 22:56:04.070057 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:04.070058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:04.070077 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:04.070128 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:04.070247 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:04.070955 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:04.071869 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:04.072174 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:04.072257 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:04.072818 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:04.072819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:04.072820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:04.072823 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:04.072837 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:04.072841 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:04.072895 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:04.072937 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 22:56:04.072961 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:56:04.081153 24453 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:04.081938 24453 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:04.089509 24453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbedf10b142249d5804df9e83aafced7: Generated new TSK 0
I20260620 22:56:04.089650 24453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:04.093107 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24365
2026-06-20T22:56:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:04 dist-test-slave-4mtw krb5kdc[24464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:04 dist-test-slave-4mtw krb5kdc[24464](info): set up 2 sockets
Jun 20 22:56:04 dist-test-slave-4mtw krb5kdc[24464](info): commencing operation
krb5kdc: starting...
W20260620 22:56:05.028564 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:05 dist-test-slave-4mtw krb5kdc[24464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:05.132992 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41657
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:05.211685 24477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:05.211880 24477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:05.211900 24477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:05.213022 24477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:05.213061 24477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:05.213074 24477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:05.213085 24477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:05.213096 24477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:05.214488 24477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:41657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:05.214663 24477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:05.214833 24477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:05.219179 24485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:05.219313 24477 server_base.cc:1034] running on GCE node
W20260620 22:56:05.219180 24483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:05.219172 24482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:05.219660 24477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:05.219931 24477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:05.221064 24477 hybrid_clock.cc:648] HybridClock initialized: now 1781996165221047 us; error 30 us; skew 500 ppm
Jun 20 22:56:05 dist-test-slave-4mtw krb5kdc[24464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:05.223278 24477 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:05.223668 24477 webserver.cc:460] Webserver started at http://127.2.90.190:42823/ using document root <none> and password file <none>
I20260620 22:56:05.223865 24477 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:05.223909 24477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:05.224023 24477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:05.225018 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "773adab21ff14f8fa927f90978a5125d"
format_stamp: "Formatted at 2026-06-20 22:56:05 on dist-test-slave-4mtw"
server_key: "64742b9abd5988a5fdd5f1aad4dbb1ba"
server_key_iv: "b6fdc4bb9cb12f50239b1464cb9592b0"
server_key_version: "encryptionkey@0"
I20260620 22:56:05.225319 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "773adab21ff14f8fa927f90978a5125d"
format_stamp: "Formatted at 2026-06-20 22:56:05 on dist-test-slave-4mtw"
server_key: "64742b9abd5988a5fdd5f1aad4dbb1ba"
server_key_iv: "b6fdc4bb9cb12f50239b1464cb9592b0"
server_key_version: "encryptionkey@0"
I20260620 22:56:05.226472 24477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:05.227123 24492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:05.227263 24477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:05.227339 24477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "773adab21ff14f8fa927f90978a5125d"
format_stamp: "Formatted at 2026-06-20 22:56:05 on dist-test-slave-4mtw"
server_key: "64742b9abd5988a5fdd5f1aad4dbb1ba"
server_key_iv: "b6fdc4bb9cb12f50239b1464cb9592b0"
server_key_version: "encryptionkey@0"
I20260620 22:56:05.227396 24477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:05.230489 24477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:05.232419 24477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:05.232523 24477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:05.235942 24477 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:41657
I20260620 22:56:05.235983 24544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:41657 every 8 connection(s)
I20260620 22:56:05.236245 24477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:05.236824 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24477
I20260620 22:56:05.236940 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:05.237131 2410 external_mini_cluster.cc:1428] Setting key 4e5e01b09773a28fd7ffdb80fef19b90
I20260620 22:56:05.237278 24545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:05.239434 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:05.237607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45651 (local address 127.2.90.190:41657)
0620 22:56:05.237840 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:05.237849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:05.237878 (+ 29us) server_negotiation.cc:407] Connection header received
0620 22:56:05.238341 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:05.238349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:05.238390 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:05.238532 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:05.239036 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:05.239040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:05.239041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:05.239045 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:05.239084 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:05.239099 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:05.239176 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:05.239211 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 22:56:05.239246 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":67,"threads_started":1}
I20260620 22:56:05.240206 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:05.240258 24545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d: Bootstrap starting.
I20260620 22:56:05.240839 24545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:05.241099 24545 log.cc:824] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:05.241781 24545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d: No bootstrap required, opened a new log
I20260620 22:56:05.242369 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:05.241494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52565 (local address 127.2.90.190:41657)
0620 22:56:05.241525 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:05.241527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:05.241644 (+ 117us) server_negotiation.cc:407] Connection header received
0620 22:56:05.241670 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:05.241671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:05.241675 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:05.241737 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:05.242194 (+ 457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:05.242195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:05.242195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 22:56:05.242199 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:05.242212 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 22:56:05.242216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:05.242273 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:05.242304 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 22:56:05.242315 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:56:05.243749 24545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773adab21ff14f8fa927f90978a5125d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41657 } }
I20260620 22:56:05.243873 24545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:56:05.243904 24545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773adab21ff14f8fa927f90978a5125d, State: Initialized, Role: FOLLOWER
I20260620 22:56:05.244014 24545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773adab21ff14f8fa927f90978a5125d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41657 } }
I20260620 22:56:05.244079 24545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:56:05.244107 24545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:56:05.244161 24545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:05.244633 24545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773adab21ff14f8fa927f90978a5125d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41657 } }
I20260620 22:56:05.244741 24545 leader_election.cc:302] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 773adab21ff14f8fa927f90978a5125d; no voters:
I20260620 22:56:05.244887 24545 leader_election.cc:288] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:56:05.244930 24555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:05.245083 24555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [term 1 LEADER]: Becoming Leader. State: Replica: 773adab21ff14f8fa927f90978a5125d, State: Running, Role: LEADER
I20260620 22:56:05.245102 24545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:05.245169 24555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773adab21ff14f8fa927f90978a5125d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41657 } }
I20260620 22:56:05.245539 24556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "773adab21ff14f8fa927f90978a5125d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773adab21ff14f8fa927f90978a5125d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41657 } } }
I20260620 22:56:05.245546 24557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 773adab21ff14f8fa927f90978a5125d. Latest consensus state: current_term: 1 leader_uuid: "773adab21ff14f8fa927f90978a5125d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773adab21ff14f8fa927f90978a5125d" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 41657 } } }
I20260620 22:56:05.245680 24556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:05.245698 24557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:05.246075 24564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:05.246503 24564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:05.247819 24564 catalog_manager.cc:1347] Generated new cluster ID: a3111991c6884cd99da97d763e38a767
I20260620 22:56:05.247866 24564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:05.259833 24564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:05.259986 24564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:05.265537 24564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 773adab21ff14f8fa927f90978a5125d: Generated new TSK 0
I20260620 22:56:05.265688 24564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:05.274039 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24477
2026-06-20T22:56:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1180 ms)
[----------] 18 tests from AuthTokenIssuingTest (21515 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:05 dist-test-slave-4mtw krb5kdc[24576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:05 dist-test-slave-4mtw krb5kdc[24576](info): set up 2 sockets
Jun 20 22:56:05 dist-test-slave-4mtw krb5kdc[24576](info): commencing operation
krb5kdc: starting...
W20260620 22:56:06.212448 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:06.316954 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44985
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44985,127.2.90.189:37487,127.2.90.188:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:06.392447 24589 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:56:06.392695 24589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:06.392721 24589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:06.392746 24589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:06.393885 24589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:06.393916 24589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:06.393929 24589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:06.393939 24589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:06.393954 24589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:06.395334 24589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44985,127.2.90.189:37487,127.2.90.188:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:44985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:06.395514 24589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:06.395675 24589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:06.400018 24594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:06.400064 24597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:06.400023 24595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:06.400357 24589 server_base.cc:1034] running on GCE node
I20260620 22:56:06.400556 24589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:06.400818 24589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:06.401947 24589 hybrid_clock.cc:648] HybridClock initialized: now 1781996166401936 us; error 25 us; skew 500 ppm
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:06.407776 24589 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:06.408166 24589 webserver.cc:460] Webserver started at http://127.2.90.190:39603/ using document root <none> and password file <none>
I20260620 22:56:06.408332 24589 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:06.408375 24589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:06.408468 24589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:06.409480 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "cee1352d555642ab8cc1535baf3b7d70"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "f6fe7c682e5aa3b4a1201cead9645c45"
server_key_iv: "891b9b8506c308498c34b2c8dc6fba50"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.409761 24589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "cee1352d555642ab8cc1535baf3b7d70"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "f6fe7c682e5aa3b4a1201cead9645c45"
server_key_iv: "891b9b8506c308498c34b2c8dc6fba50"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.410990 24589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 22:56:06.411566 24604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:06.411700 24589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:56:06.411777 24589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "cee1352d555642ab8cc1535baf3b7d70"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "f6fe7c682e5aa3b4a1201cead9645c45"
server_key_iv: "891b9b8506c308498c34b2c8dc6fba50"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.411839 24589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:06.428541 24589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:06.430188 24589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:06.430291 24589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:06.433915 24589 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:44985
I20260620 22:56:06.433964 24656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:44985 every 8 connection(s)
I20260620 22:56:06.434259 24589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:06.435158 24657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:06.436599 24657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:06.440646 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24589
I20260620 22:56:06.440776 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:06.440953 2410 external_mini_cluster.cc:1428] Setting key dcd456420470899e8b0a36c0f34e766f
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:06.445339 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.437342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.2.90.190:44985)
0620 22:56:06.437448 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.437452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.437484 (+ 32us) server_negotiation.cc:407] Connection header received
0620 22:56:06.437978 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.437980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.437996 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.438068 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.438588 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.439370 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.440180 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.440475 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.440566 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.442611 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.442618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.442618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.442624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.443951 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.444267 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.444270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.444271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.444348 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.444534 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.444535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.444535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.444768 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.444835 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.445003 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.445110 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":40,"threads_started":1}
I20260620 22:56:06.445344 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.437190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44985 (local address 127.0.0.1:36322)
0620 22:56:06.437327 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.437359 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.437826 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.438079 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.438082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.438344 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.438501 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.438506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.439454 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.439456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.440103 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.440105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.440559 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.440560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.440619 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.441134 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.441144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.442533 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.444019 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.444025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.444027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.444203 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.444401 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.444401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.444402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.444484 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.444821 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.444826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.444887 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.444968 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.445181 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":81,"thread_start_us":67,"threads_started":1}
I20260620 22:56:06.445950 24657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:06.446245 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.446144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50910)
0620 22:56:06.446180 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.446222 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 22:56:06.446368 24606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
I20260620 22:56:06.446709 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.446631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50916)
0620 22:56:06.446665 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.446687 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 22:56:06.446798 24657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:37487: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.90.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab.
Entry for principal kudu/127.2.90.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab.
I20260620 22:56:06.461056 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:37487
--webserver_interface=127.2.90.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab
--principal=kudu/127.2.90.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44985,127.2.90.189:37487,127.2.90.188:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:56:06.496016 24657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } attempt: 1
I20260620 22:56:06.496486 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.496363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50924)
0620 22:56:06.496422 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.496457 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 22:56:06.496867 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.496785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50930)
0620 22:56:06.496825 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.496849 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:56:06.496963 24657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:37487: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
W20260620 22:56:06.538296 24664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:56:06.538508 24664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:06.538530 24664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:06.538555 24664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:06.539628 24664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:06.539665 24664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:06.539683 24664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:06.539700 24664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:06.539716 24664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:06.541177 24664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44985,127.2.90.189:37487,127.2.90.188:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab
--principal=kudu/127.2.90.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.90.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:06.541414 24664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:06.541599 24664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:06.545944 24669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:06.546036 24664 server_base.cc:1034] running on GCE node
W20260620 22:56:06.545945 24672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:06.546113 24670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:06.546401 24664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:06.546666 24664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:06.547802 24664 hybrid_clock.cc:648] HybridClock initialized: now 1781996166547780 us; error 35 us; skew 500 ppm
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:06.550102 24664 init.cc:377] Logged in from keytab as kudu/127.2.90.189@KRBTEST.COM (short username kudu)
I20260620 22:56:06.550495 24664 webserver.cc:460] Webserver started at http://127.2.90.189:38639/ using document root <none> and password file <none>
I20260620 22:56:06.550685 24664 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:06.550725 24664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:06.550843 24664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:06.551820 24664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data/instance:
uuid: "5f5b60237a24414eb4798f05b7dcaf18"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "4a1a30f9de6bada500ee75492a6e9d17"
server_key_iv: "c6ecb1a54b7739b2aa31848c392352f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.552132 24664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal/instance:
uuid: "5f5b60237a24414eb4798f05b7dcaf18"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "4a1a30f9de6bada500ee75492a6e9d17"
server_key_iv: "c6ecb1a54b7739b2aa31848c392352f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.553344 24664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 22:56:06.553938 24679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:06.554085 24664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 22:56:06.554162 24664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal
uuid: "5f5b60237a24414eb4798f05b7dcaf18"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "4a1a30f9de6bada500ee75492a6e9d17"
server_key_iv: "c6ecb1a54b7739b2aa31848c392352f7"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.554212 24664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:06.565088 24657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } attempt: 2
I20260620 22:56:06.565455 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.565340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50942)
0620 22:56:06.565389 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.565417 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:56:06.565763 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.565698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50946)
0620 22:56:06.565728 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.565748 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 22:56:06.565840 24657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:37487: Network error: Client connection negotiation failed: client connection to 127.2.90.189:37487: connect: Connection refused (error 111)
I20260620 22:56:06.570182 24664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:06.571779 24664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:06.571885 24664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:06.575403 24664 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.189:37487
I20260620 22:56:06.575443 24731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.189:37487 every 8 connection(s)
I20260620 22:56:06.575726 24664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
I20260620 22:56:06.576611 24732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:06.578049 24732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:06.585155 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24664
I20260620 22:56:06.585263 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-1/wal/instance
I20260620 22:56:06.585427 2410 external_mini_cluster.cc:1428] Setting key 60301ad3f441878f2ac45f630044b73d
I20260620 22:56:06.585852 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.578627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.2.90.190:44985)
0620 22:56:06.578670 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.578671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.578837 (+ 166us) server_negotiation.cc:407] Connection header received
0620 22:56:06.579288 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.579289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.579304 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.579354 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.579920 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.580389 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.581327 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.581534 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.581589 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.583415 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.583417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.583417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.583425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.584594 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.584965 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.584966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.584966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.585017 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.585223 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.585223 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.585224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.585348 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.585402 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.585686 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.585766 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 22:56:06.585988 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.578604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44985 (local address 127.0.0.1:36328)
0620 22:56:06.578732 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.578765 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.579199 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.579416 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.579419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.579653 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.579833 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.579836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.580466 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.580467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.581252 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.581254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.581616 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.581618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.581718 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.582137 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.582146 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.583332 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.584663 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.584670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.584672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.584899 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.585067 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.585068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.585068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.585169 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.585405 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.585411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.585565 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.585657 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.585794 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260620 22:56:06.586428 24732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.90.188@KRBTEST.COM; defaulting to no policy
I20260620 22:56:06.592590 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.586629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50952)
0620 22:56:06.586664 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.586673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.586703 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.586913 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.586914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.587031 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.587068 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.587070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.587816 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.587817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.588352 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.588354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.588795 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.588796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.588852 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.589166 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.589171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.590309 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.591848 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.591849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.591849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.591989 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.592136 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.592136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.592137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.592177 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.592382 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.592383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.592424 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.592472 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.592532 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 22:56:06.592712 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.586703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.2.90.189:37487)
0620 22:56:06.586811 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.586814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.586827 (+ 13us) server_negotiation.cc:407] Connection header received
0620 22:56:06.586841 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.586842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.586850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.586902 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.587127 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.587748 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.588423 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.588742 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.588795 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.590386 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.590393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.590393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.590399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.591782 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.592046 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.592048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.592048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.592094 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.592210 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.592210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.592210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.592329 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.592381 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.592515 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.592578 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260620 22:56:06.593196 24732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:06.593482 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.593396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50082)
0620 22:56:06.593431 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.593467 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:56:06.593585 24681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.188@KRBTEST.COM" created.
I20260620 22:56:06.593962 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.593886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50098)
0620 22:56:06.593920 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.593942 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 22:56:06.594058 24732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:42423: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab.
Entry for principal kudu/127.2.90.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab.
I20260620 22:56:06.605067 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:42423
--webserver_interface=127.2.90.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab
--principal=kudu/127.2.90.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:44985,127.2.90.189:37487,127.2.90.188:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:56:06.643285 24732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } attempt: 1
I20260620 22:56:06.643766 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.643641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50102)
0620 22:56:06.643696 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.643737 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 22:56:06.644140 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.644027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50110)
0620 22:56:06.644075 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.644105 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:56:06.644241 24732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:42423: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
I20260620 22:56:06.656976 24657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } attempt: 3
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:56:06.664000 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.657398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.2.90.189:37487)
0620 22:56:06.657439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.657441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.657453 (+ 12us) server_negotiation.cc:407] Connection header received
0620 22:56:06.657535 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.657536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.657546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.657604 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.657972 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.658458 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.659441 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.659651 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.659695 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.661704 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.661705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.661706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.661713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.663028 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.663378 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.663379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.663379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.663425 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.663599 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.663599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.663599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.663693 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.663735 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.663877 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.663932 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:56:06.664053 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.657335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50968)
0620 22:56:06.657396 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.657407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.657468 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.657622 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.657623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.657820 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.657892 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.657895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.658534 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.658535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.659361 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.659364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.659713 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.659714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.659790 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.660208 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.660212 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.661614 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.663104 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.663106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.663106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.663316 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.663478 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.663479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.663479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.663545 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.663741 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.663742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.663791 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.663852 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.663956 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 22:56:06.664484 24657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:06.664819 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.664720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50122)
0620 22:56:06.664757 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.664791 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 22:56:06.665103 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.665023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50130)
0620 22:56:06.665063 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.665086 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:56:06.665184 24657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:42423: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
W20260620 22:56:06.683969 24739 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:56:06.684154 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:06.684170 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:06.684199 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:06.685320 24739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:06.685357 24739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:06.685370 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:06.685388 24739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:06.685410 24739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:06.686810 24739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:44985,127.2.90.189:37487,127.2.90.188:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab
--principal=kudu/127.2.90.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.90.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:06.687021 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:06.687209 24739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:06.691461 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:06.691499 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:06.691498 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:06.691641 24739 server_base.cc:1034] running on GCE node
I20260620 22:56:06.691838 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:06.692083 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:06.693222 24739 hybrid_clock.cc:648] HybridClock initialized: now 1781996166693192 us; error 43 us; skew 500 ppm
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:06.695451 24739 init.cc:377] Logged in from keytab as kudu/127.2.90.188@KRBTEST.COM (short username kudu)
I20260620 22:56:06.695847 24739 webserver.cc:460] Webserver started at http://127.2.90.188:33255/ using document root <none> and password file <none>
I20260620 22:56:06.696033 24739 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:06.696072 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:06.696187 24739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:06.696316 24657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } attempt: 1
I20260620 22:56:06.696767 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.696600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50144)
0620 22:56:06.696669 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.696725 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 22:56:06.697055 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.696966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50146)
0620 22:56:06.697009 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.697034 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 22:56:06.697141 24657 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:42423: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
I20260620 22:56:06.697399 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data/instance:
uuid: "ab26df4305da4f088733fe1f47d20e92"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "a238ea533fca333485c95880b8e235d7"
server_key_iv: "08e53ad9cb78511c09b126c2513af081"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.697714 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal/instance:
uuid: "ab26df4305da4f088733fe1f47d20e92"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "a238ea533fca333485c95880b8e235d7"
server_key_iv: "08e53ad9cb78511c09b126c2513af081"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.698908 24739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:06.699544 24754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:06.699671 24739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:06.699743 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal
uuid: "ab26df4305da4f088733fe1f47d20e92"
format_stamp: "Formatted at 2026-06-20 22:56:06 on dist-test-slave-4mtw"
server_key: "a238ea533fca333485c95880b8e235d7"
server_key_iv: "08e53ad9cb78511c09b126c2513af081"
server_key_version: "encryptionkey@0"
I20260620 22:56:06.699800 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:06.712376 24732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } attempt: 2
I20260620 22:56:06.712755 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.712599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50154)
0620 22:56:06.712655 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.712712 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 22:56:06.713044 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.712979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50168)
0620 22:56:06.713010 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.713030 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 22:56:06.713121 24732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:42423: Network error: Client connection negotiation failed: client connection to 127.2.90.188:42423: connect: Connection refused (error 111)
I20260620 22:56:06.715097 24739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:06.717116 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:06.717236 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:06.720750 24739 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.188:42423
I20260620 22:56:06.720789 24806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.188:42423 every 8 connection(s)
I20260620 22:56:06.721086 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
I20260620 22:56:06.721920 24807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:06.723331 24807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:06.729074 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24739
I20260620 22:56:06.729182 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781995912840861-2410-0/minicluster-data/master-2/wal/instance
I20260620 22:56:06.729349 2410 external_mini_cluster.cc:1428] Setting key 8812c07915e0191eafe372aa92c81ffd
I20260620 22:56:06.731592 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.723930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.2.90.190:44985)
0620 22:56:06.723970 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.723972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.724143 (+ 171us) server_negotiation.cc:407] Connection header received
0620 22:56:06.724625 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.724627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.724639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.724705 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.725294 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.725827 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.726752 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.726944 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.726998 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.728950 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.728952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.728952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.728959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.730412 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.730796 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.730796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.730797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.730845 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.731074 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.731074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.731074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.731163 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.731205 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.731475 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.731530 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:06.731792 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.723900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:44985 (local address 127.0.0.1:36332)
0620 22:56:06.724036 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.724070 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.724533 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.724790 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.724794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.725024 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.725209 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.725214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.725907 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.725908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.726675 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.726677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.727026 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.727027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.727134 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.727549 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.727557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.728853 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.730502 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.730506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.730507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.730730 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.730896 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.730897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.730897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.731020 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.731209 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.731213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.731350 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.731448 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.731584 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260620 22:56:06.732333 24807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:56:06.737686 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.729683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.2.90.188:42423)
0620 22:56:06.729867 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.729871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.729917 (+ 46us) server_negotiation.cc:407] Connection header received
0620 22:56:06.729992 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.729993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.730003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.730058 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.730681 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.731625 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.732494 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.732943 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.732997 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.735137 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.735143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.735143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.735148 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.736581 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.736951 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.736953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.736954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.737011 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.737174 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.737175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.737175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.737334 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.737477 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.737519 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.737585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
I20260620 22:56:06.738492 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.732533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:37487 (local address 127.0.0.1:50974)
0620 22:56:06.732576 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.732585 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.732615 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.732739 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.732739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.732868 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.732918 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.732919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.733684 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.733685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.734144 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.734145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.734471 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.734472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.734540 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.734846 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.734850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.736063 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.737571 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.737572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.737573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.737735 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.737946 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.737947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.737948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.738003 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.738227 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.738227 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.738266 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.738323 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.738403 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":3}
I20260620 22:56:06.738507 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.732584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.2.90.189:37487)
0620 22:56:06.732624 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.732626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.732633 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:06.732658 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.732658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.732667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.732728 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.732978 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.733614 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.734221 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.734409 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.734458 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.736137 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.736138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.736138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.736146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.737500 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.737832 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.737833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.737833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.737889 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.738056 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.738056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.738057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.738174 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.738224 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.738352 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.738414 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:56:06.739123 24807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:56:06.744848 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.738390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.2.90.190:44985)
0620 22:56:06.738425 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.738426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.738432 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:56:06.738454 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.738454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.738462 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.738506 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.738935 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.739592 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.740228 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.740398 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.740448 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.742483 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.742484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.742485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.742491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.743855 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.744169 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.744170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.744170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.744224 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.744384 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.744385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.744385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.744483 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.744629 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.744663 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.744747 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 22:56:06.744863 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.739330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50182)
0620 22:56:06.739365 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.739375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.739406 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.739513 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.739514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.739605 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.739634 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.739635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.740176 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.740176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.740838 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.740841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.741155 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.741156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.741211 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.741496 (+ 285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.741500 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.742798 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.743967 (+ 1169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.743968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.743968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.744103 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.744260 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.744260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.744261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.744323 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.744533 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.744534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.744570 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.744630 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.744751 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 22:56:06.745080 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.739358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.2.90.188:42423)
0620 22:56:06.739388 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.739389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.739397 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:06.739446 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.739447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.739455 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.739510 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.739691 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.740118 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.740903 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.741093 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.741140 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.742857 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.742858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.742858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.742864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.743899 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.744159 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.744160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.744160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.744209 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.744375 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.744375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.744376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.744480 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.744531 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.744659 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.744800 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":14}
I20260620 22:56:06.746476 24807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: Bootstrap starting.
I20260620 22:56:06.747424 24807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:06.747696 24807 log.cc:824] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:06.748538 24807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: No bootstrap required, opened a new log
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:56:06.750344 24807 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.750463 24807 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab26df4305da4f088733fe1f47d20e92, State: Initialized, Role: FOLLOWER
I20260620 22:56:06.750523 24807 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.750835 24807 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:06.751535 24813 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.751699 24813 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:06.752434 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.745412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.2.90.189:37487)
0620 22:56:06.745457 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.745458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.745468 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:06.745487 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.745488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.745497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.745548 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.745916 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.746563 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.747434 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.747671 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.747717 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.749979 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.749980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.749981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.749989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.751388 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.751715 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.751716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.751716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.751774 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.751974 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.751974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.751974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.752070 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.752210 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.752283 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.752349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 22:56:06.752856 24824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:06.752959 24824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:06.753942 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:06.754838 24631 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cee1352d555642ab8cc1535baf3b7d70"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36338
I20260620 22:56:06.753401 24739 master_runner.cc:186] Error getting master registration for 127.2.90.190:44985: OK, instance_id { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" instance_seqno: 1781996166432931 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:56:06.756153 24739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:56:06.772286 24657 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } attempt: 2
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:56:06.779939 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.772716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.2.90.188:42423)
0620 22:56:06.772760 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.772762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.772799 (+ 37us) server_negotiation.cc:407] Connection header received
0620 22:56:06.772897 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.772898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.772909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.772970 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.773290 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.774048 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.774994 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.775291 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.775347 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.777471 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.777473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.777473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.777481 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.778941 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.779222 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.779224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.779224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.779277 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.779442 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.779442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.779443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.779584 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.779640 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.779785 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.779850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:06.779939 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.772641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:50186)
0620 22:56:06.772739 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.772751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.772816 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.772987 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.772988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.773169 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.773225 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.773227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.774117 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.774118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.774914 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.774917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.775348 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.775349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.775403 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.775885 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.775890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.777379 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.779002 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.779003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.779004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.779163 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.779323 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.779323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.779324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.779392 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.779645 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.779646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.779692 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.779750 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.779840 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 22:56:06.781298 24657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70: Bootstrap starting.
I20260620 22:56:06.781888 24657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:06.782141 24657 log.cc:824] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:06.782812 24657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70: No bootstrap required, opened a new log
I20260620 22:56:06.783779 24657 raft_consensus.cc:348] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.783897 24657 raft_consensus.cc:729] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee1352d555642ab8cc1535baf3b7d70, State: Initialized, Role: FOLLOWER
I20260620 22:56:06.783972 24657 consensus_queue.cc:260] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.784210 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.784273 24657 sys_catalog.cc:564] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:06.784284 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:06.784408 24631 raft_consensus.cc:482] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:56:06.784518 24631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:06.785342 24631 raft_consensus.cc:504] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
W20260620 22:56:06.786008 24838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:06.786095 24838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:06.786324 24631 leader_election.cc:288] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [CANDIDATE]: Term 1 election: Requested vote from peers 5f5b60237a24414eb4798f05b7dcaf18 (127.2.90.189:37487), ab26df4305da4f088733fe1f47d20e92 (127.2.90.188:42423)
I20260620 22:56:06.786629 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cee1352d555642ab8cc1535baf3b7d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ab26df4305da4f088733fe1f47d20e92"
I20260620 22:56:06.786753 24781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:06.787581 24781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee1352d555642ab8cc1535baf3b7d70 in term 1.
I20260620 22:56:06.788780 24606 leader_election.cc:302] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab26df4305da4f088733fe1f47d20e92, cee1352d555642ab8cc1535baf3b7d70; no voters:
I20260620 22:56:06.788890 24827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:06.789012 24827 raft_consensus.cc:686] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [term 1 LEADER]: Becoming Leader. State: Replica: cee1352d555642ab8cc1535baf3b7d70, State: Running, Role: LEADER
I20260620 22:56:06.789086 24827 consensus_queue.cc:237] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.789484 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee1352d555642ab8cc1535baf3b7d70. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.789547 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:06.789906 24589 master_runner.cc:186] Error getting master registration for 127.2.90.189:37487: OK, instance_id { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" instance_seqno: 1781996166574437 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:56:06.792047 24589 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:56:06.793206 24849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:06.793375 24849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:06.797422 24736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.788132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.2.90.189:37487)
0620 22:56:06.788352 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.788354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.788361 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:06.788381 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.788382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.788392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.788449 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.789481 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.790240 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.791726 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.792043 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.792081 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.794241 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.794242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.794243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.794249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.795879 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.796420 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.796421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.796422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.796484 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.796758 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.796759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.796760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.796887 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.796963 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.797257 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.797331 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260620 22:56:06.804190 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.787585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.2.90.188:42423)
0620 22:56:06.787620 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.787621 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.792280 (+ 4659us) server_negotiation.cc:407] Connection header received
0620 22:56:06.792374 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.792375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.792384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.792411 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.793115 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.793767 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.795601 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.795882 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.795921 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.799477 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.799478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.799478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.799485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.801189 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.801727 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.801728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.801728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.801782 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.803540 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.803541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.803541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.803666 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.803746 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.804024 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.804096 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:06.804252 24732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } attempt: 3
I20260620 22:56:06.804771 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.788312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.2.90.190:44985)
0620 22:56:06.788648 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.788650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.794815 (+ 6165us) server_negotiation.cc:407] Connection header received
0620 22:56:06.794902 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.794903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.794911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.794938 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.797051 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.797742 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.799743 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.800034 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.800069 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.801741 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.801743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.801743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.801750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.803316 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.803790 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.803791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.803792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.803840 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.804006 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.804007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.804007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.804117 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.804195 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.804604 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.804694 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260620 22:56:06.805804 24781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [term 1 FOLLOWER]: Refusing update from remote peer cee1352d555642ab8cc1535baf3b7d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:06.806118 24827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:06.807943 24813 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee1352d555642ab8cc1535baf3b7d70. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.808025 24813 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 22:56:06 dist-test-slave-4mtw krb5kdc[24576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996166, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:56:06.809139 24847 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.810071 24813 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.810144 24813 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:06.810204 24847 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:06.810395 24850 mvcc.cc:205] Tried to move back new op lower bound from 7299056299234058240 to 7299056299168858112. Current Snapshot: MvccSnapshot[applied={T|T < 7299056299234058240}]
I20260620 22:56:06.810472 24851 mvcc.cc:205] Tried to move back new op lower bound from 7299056299234058240 to 7299056299168858112. Current Snapshot: MvccSnapshot[applied={T|T < 7299056299234058240}]
I20260620 22:56:06.811268 24849 catalog_manager.cc:1347] Generated new cluster ID: 4451965a69024775b8248bc5fada0dda
I20260620 22:56:06.811311 24849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:06.811805 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.804460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:42423 (local address 127.0.0.1:33612)
0620 22:56:06.804528 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:06.804538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:06.804569 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:06.804766 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:06.804767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:06.804895 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:06.804930 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.804932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.805970 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.805971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.806663 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:06.806666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.807084 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.807085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:06.807145 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.807499 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:06.807503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:06.808954 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:06.810665 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.810666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.810666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.810872 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.811112 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:06.811112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:06.811113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:06.811180 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:06.811515 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:06.811516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:06.811570 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 22:56:06.811628 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 22:56:06.811711 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 22:56:06.812382 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:06.804534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.2.90.188:42423)
0620 22:56:06.804649 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.804651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.804657 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:56:06.804672 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.804673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.804702 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.804756 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.805002 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.805694 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.806763 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:06.807015 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:06.807083 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:06.809038 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:06.809039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:06.809040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:06.809046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:06.810468 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.810972 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.810973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.810973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.811037 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:06.811252 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:06.811253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:06.811253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:06.811378 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:06.811430 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:06.812218 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 22:56:06.812301 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 22:56:06.813706 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: Bootstrap starting.
I20260620 22:56:06.814339 24732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:06.814604 24732 log.cc:824] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:06.821267 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: No bootstrap required, opened a new log
I20260620 22:56:06.823217 24732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.823338 24732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5b60237a24414eb4798f05b7dcaf18, State: Initialized, Role: FOLLOWER
I20260620 22:56:06.823406 24732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } }
I20260620 22:56:06.823696 24732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:06.824874 24705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:06.825287 24849 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:06.825767 24705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [term 1 FOLLOWER]: Refusing update from remote peer cee1352d555642ab8cc1535baf3b7d70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:06.825964 24849 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:06.826048 24827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:06.826352 24706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cee1352d555642ab8cc1535baf3b7d70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f5b60237a24414eb4798f05b7dcaf18"
I20260620 22:56:06.827097 24706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee1352d555642ab8cc1535baf3b7d70 in term 1.
I20260620 22:56:06.831174 24856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.831293 24856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:06.831350 24856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee1352d555642ab8cc1535baf3b7d70. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.831542 24856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:06.831584 24856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.831619 24856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:06.833215 24866 mvcc.cc:205] Tried to move back new op lower bound from 7299056299310792704 to 7299056299168858112. Current Snapshot: MvccSnapshot[applied={T|T < 7299056299234058240}]
W20260620 22:56:06.834359 24871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:06.834510 24871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:06.834764 24827 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cee1352d555642ab8cc1535baf3b7d70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee1352d555642ab8cc1535baf3b7d70" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 44985 } } peers { permanent_uuid: "5f5b60237a24414eb4798f05b7dcaf18" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 37487 } } peers { permanent_uuid: "ab26df4305da4f088733fe1f47d20e92" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 42423 } } }
I20260620 22:56:06.834849 24827 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:06.836328 24849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cee1352d555642ab8cc1535baf3b7d70: Generated new TSK 0
I20260620 22:56:06.836490 24849 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 22:56:06.859478 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.857992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.2.90.189:37487)
0620 22:56:06.858033 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.858035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.858859 (+ 824us) server_negotiation.cc:407] Connection header received
0620 22:56:06.858968 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.858970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.858980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.859009 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.859409 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44272: BlockingRecv error: recv got EOF from 127.0.0.1:44272 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:56:06.860173 24810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.858331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.2.90.188:42423)
0620 22:56:06.858370 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.858371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.859696 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 22:56:06.859785 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.859786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.859796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.859822 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.860114 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33614: BlockingRecv error: recv got EOF from 127.0.0.1:33614 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:56:06.860880 24660 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.858636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.2.90.190:44985)
0620 22:56:06.858678 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.858680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.860378 (+ 1698us) server_negotiation.cc:407] Connection header received
0620 22:56:06.860467 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.860468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.860478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.860505 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.860820 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56704: BlockingRecv error: recv got EOF from 127.0.0.1:56704 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:56:06.863059 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.862346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.2.90.189:37487)
0620 22:56:06.862385 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.862386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.862579 (+ 193us) server_negotiation.cc:407] Connection header received
0620 22:56:06.862674 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.862675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.862692 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.862719 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.863016 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44276: BlockingRecv error: recv got EOF from 127.0.0.1:44276 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:56:06.865885 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.865211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.2.90.189:37487)
0620 22:56:06.865240 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.865242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.865445 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:56:06.865531 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.865532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.865540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.865565 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.865845 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44288: BlockingRecv error: recv got EOF from 127.0.0.1:44288 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 22:56:06.869606 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.868946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.2.90.189:37487)
0620 22:56:06.868975 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.868977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.869178 (+ 201us) server_negotiation.cc:407] Connection header received
0620 22:56:06.869262 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.869263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.869271 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.869297 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.869566 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44302: BlockingRecv error: recv got EOF from 127.0.0.1:44302 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 22:56:06.875358 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.874647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.2.90.189:37487)
0620 22:56:06.874678 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.874680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.874882 (+ 202us) server_negotiation.cc:407] Connection header received
0620 22:56:06.874983 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.874984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.874993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.875019 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.875314 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44316: BlockingRecv error: recv got EOF from 127.0.0.1:44316 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:56:06.885448 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.884604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.2.90.189:37487)
0620 22:56:06.884639 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.884640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.884882 (+ 242us) server_negotiation.cc:407] Connection header received
0620 22:56:06.884991 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.884992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.885002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.885030 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.885354 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44326: BlockingRecv error: recv got EOF from 127.0.0.1:44326 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:56:06.903694 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.902801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.2.90.189:37487)
0620 22:56:06.902853 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.902855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.903128 (+ 273us) server_negotiation.cc:407] Connection header received
0620 22:56:06.903228 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.903229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.903240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.903267 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.903634 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44330: BlockingRecv error: recv got EOF from 127.0.0.1:44330 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 22:56:06.937915 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:06.937164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.2.90.189:37487)
0620 22:56:06.937207 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:06.937209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:06.937341 (+ 132us) server_negotiation.cc:407] Connection header received
0620 22:56:06.937404 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:06.937405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:06.937416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:06.937491 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:06.937855 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44342: BlockingRecv error: recv got EOF from 127.0.0.1:44342 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 22:56:07.003556 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:07.002899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.2.90.189:37487)
0620 22:56:07.002952 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.002954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.003059 (+ 105us) server_negotiation.cc:407] Connection header received
0620 22:56:07.003115 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.003116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.003129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.003177 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:07.003495 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44350: BlockingRecv error: recv got EOF from 127.0.0.1:44350 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 22:56:07.133266 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:07.132608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.2.90.189:37487)
0620 22:56:07.132702 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.132704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.132770 (+ 66us) server_negotiation.cc:407] Connection header received
0620 22:56:07.132842 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.132843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.132856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.132901 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:07.133216 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44366: BlockingRecv error: recv got EOF from 127.0.0.1:44366 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 22:56:07.390978 24736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:07.390341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.2.90.189:37487)
0620 22:56:07.390398 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.390401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.390479 (+ 78us) server_negotiation.cc:407] Connection header received
0620 22:56:07.390540 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.390541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.390553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.390598 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:07.390916 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44374: BlockingRecv error: recv got EOF from 127.0.0.1:44374 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 22:56:07.753489 24824 catalog_manager.cc:1259] Loaded cluster ID: 4451965a69024775b8248bc5fada0dda
I20260620 22:56:07.753562 24824 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: loading cluster ID for follower catalog manager: success
I20260620 22:56:07.754375 24824 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: acquiring CA information for follower catalog manager: success
I20260620 22:56:07.754675 24824 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab26df4305da4f088733fe1f47d20e92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:56:07.834949 24871 catalog_manager.cc:1259] Loaded cluster ID: 4451965a69024775b8248bc5fada0dda
I20260620 22:56:07.835026 24871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: loading cluster ID for follower catalog manager: success
I20260620 22:56:07.835829 24871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: acquiring CA information for follower catalog manager: success
I20260620 22:56:07.836144 24871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f5b60237a24414eb4798f05b7dcaf18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:56:07.906198 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:07.903972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.2.90.189:37487)
0620 22:56:07.904073 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.904074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.904084 (+ 10us) server_negotiation.cc:407] Connection header received
0620 22:56:07.904151 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.904152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.904202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.904272 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:07.904389 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.904912 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.905605 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.905786 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.905829 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:07.905934 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:07.906036 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:07.906088 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:07.906103 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 22:56:07.906118 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
I20260620 22:56:07.909950 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:07.907875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.2.90.188:42423)
0620 22:56:07.907986 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.907988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.908018 (+ 30us) server_negotiation.cc:407] Connection header received
0620 22:56:07.908087 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.908088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.908121 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.908177 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:07.908249 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.908756 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.909402 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.909579 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.909620 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:07.909713 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:07.909807 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:07.909854 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:07.909869 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 22:56:07.909883 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260620 22:56:07.914029 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:07.911425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.2.90.189:37487)
0620 22:56:07.911478 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.911480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.911575 (+ 95us) server_negotiation.cc:407] Connection header received
0620 22:56:07.911611 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.911612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.911622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.911661 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:07.911823 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.912499 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.913295 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.913589 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.913646 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:07.913778 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:07.913818 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:07.913862 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:07.913937 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 22:56:07.913960 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:56:07.914057 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:07.911384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.2.90.190:44985)
0620 22:56:07.911509 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.911511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.911520 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:07.911566 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.911567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.911615 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.911711 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:07.911808 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.912360 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.913235 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.913475 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.913530 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:07.913758 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:07.913858 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:07.913912 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:07.913957 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 22:56:07.913979 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":43,"threads_started":1}
I20260620 22:56:07.914552 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:07.911611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.2.90.188:42423)
0620 22:56:07.911650 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:07.911652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:07.911759 (+ 107us) server_negotiation.cc:407] Connection header received
0620 22:56:07.911782 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:07.911783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:07.911792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:07.911840 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:07.911995 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.912712 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.913680 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:07.913941 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:07.913992 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:07.914152 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:07.914182 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:07.914306 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:07.914460 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 22:56:07.914482 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:56:07.915647 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24589
I20260620 22:56:07.920392 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24664
I20260620 22:56:07.924451 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24739
2026-06-20T22:56:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2651 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 22:56:07 dist-test-slave-4mtw krb5kdc[24957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 22:56:07 dist-test-slave-4mtw krb5kdc[24957](info): set up 2 sockets
Jun 20 22:56:07 dist-test-slave-4mtw krb5kdc[24957](info): commencing operation
krb5kdc: starting...
W20260620 22:56:08.868472 2410 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/test-user.keytab.
Jun 20 22:56:08 dist-test-slave-4mtw krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T22:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.90.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
Entry for principal kudu/127.2.90.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab.
I20260620 22:56:08.970626 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:36443
--webserver_interface=127.2.90.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:36443,127.2.90.189:40635,127.2.90.188:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 22:56:09.047966 24970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:56:09.048158 24970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:09.048174 24970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:09.048192 24970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:09.049297 24970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:09.049341 24970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:09.049355 24970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:09.049365 24970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:09.049376 24970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:09.050742 24970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:36443,127.2.90.189:40635,127.2.90.188:41023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.190.keytab
--principal=kudu/127.2.90.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.190:36443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.90.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:09.050917 24970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:09.051080 24970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:09.055562 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:09.055562 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:09.055630 24970 server_base.cc:1034] running on GCE node
W20260620 22:56:09.055569 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:09.056017 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:09.056322 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:09.057468 24970 hybrid_clock.cc:648] HybridClock initialized: now 1781996169057448 us; error 34 us; skew 500 ppm
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:09.059696 24970 init.cc:377] Logged in from keytab as kudu/127.2.90.190@KRBTEST.COM (short username kudu)
I20260620 22:56:09.060058 24970 webserver.cc:460] Webserver started at http://127.2.90.190:43107/ using document root <none> and password file <none>
I20260620 22:56:09.060220 24970 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:09.060256 24970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:09.060339 24970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:09.061329 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data/instance:
uuid: "358b3abe3b414490871f0e98aed06f55"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "ed593af91f10a369465aee290827237c"
server_key_iv: "0aa5f89b9f5d91f6eaa186cb5887d416"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.061625 24970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal/instance:
uuid: "358b3abe3b414490871f0e98aed06f55"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "ed593af91f10a369465aee290827237c"
server_key_iv: "0aa5f89b9f5d91f6eaa186cb5887d416"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.062762 24970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:09.063344 24985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:09.063482 24970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:09.063560 24970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal
uuid: "358b3abe3b414490871f0e98aed06f55"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "ed593af91f10a369465aee290827237c"
server_key_iv: "0aa5f89b9f5d91f6eaa186cb5887d416"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.063619 24970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:09.081205 24970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:09.082775 24970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:09.082903 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:09.086397 24970 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.190:36443
I20260620 22:56:09.086429 25037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.190:36443 every 8 connection(s)
I20260620 22:56:09.086728 24970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/data/info.pb
I20260620 22:56:09.087596 25038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:09.089061 25038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:09.094448 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 24970
I20260620 22:56:09.094543 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-0/wal/instance
I20260620 22:56:09.094730 2410 external_mini_cluster.cc:1428] Setting key c77310d3353a89436c70c403220d0956
I20260620 22:56:09.097461 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.089645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:36443 (local address 127.0.0.1:32860)
0620 22:56:09.089801 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.089835 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.090261 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.090491 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.090493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.090713 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.090859 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.090863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.091775 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.091776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.092429 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.092430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.092916 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.092917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.093000 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.093401 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.093408 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.094757 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.096153 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.096158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.096160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.096339 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.096537 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.096538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.096538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.096618 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.096971 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.096975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.097036 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.097121 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.097328 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":64,"thread_start_us":78,"threads_started":1}
I20260620 22:56:09.097465 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.089678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.2.90.190:36443)
0620 22:56:09.089860 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.089864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.089890 (+ 26us) server_negotiation.cc:407] Connection header received
0620 22:56:09.090390 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.090391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.090407 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.090475 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.090923 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.091707 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.092488 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.092813 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.092887 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.094840 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.094847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.094847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.094854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.096079 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.096406 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.096409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.096409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.096486 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.096672 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.096673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.096673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.096910 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.096979 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.097167 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.097258 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":59,"threads_started":1}
I20260620 22:56:09.098321 25038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:09.098614 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.098500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40226)
0620 22:56:09.098544 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.098590 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:56:09.098750 24989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
I20260620 22:56:09.099138 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.099045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40228)
0620 22:56:09.099086 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.099116 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:56:09.099233 25038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:40635: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.90.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab.
Entry for principal kudu/127.2.90.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab.
I20260620 22:56:09.116456 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:40635
--webserver_interface=127.2.90.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab
--principal=kudu/127.2.90.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:36443,127.2.90.189:40635,127.2.90.188:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:56:09.148464 25038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } attempt: 1
I20260620 22:56:09.148984 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.148838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40234)
0620 22:56:09.148903 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.148949 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 22:56:09.149374 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.149252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40238)
0620 22:56:09.149324 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.149351 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:56:09.149483 25038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:40635: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
W20260620 22:56:09.200809 25045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:56:09.201018 25045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:09.201035 25045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:09.201059 25045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:09.202243 25045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:09.202286 25045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:09.202297 25045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:09.202309 25045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:09.202320 25045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:09.203749 25045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:36443,127.2.90.189:40635,127.2.90.188:41023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.189.keytab
--principal=kudu/127.2.90.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.189:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.90.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:09.203969 25045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:09.204159 25045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:09.208482 25050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:09.208484 25053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:56:09.208520 25051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:09.208621 25045 server_base.cc:1034] running on GCE node
I20260620 22:56:09.208910 25045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:09.209169 25045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:09.210297 25045 hybrid_clock.cc:648] HybridClock initialized: now 1781996169210286 us; error 27 us; skew 500 ppm
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:09.212459 25045 init.cc:377] Logged in from keytab as kudu/127.2.90.189@KRBTEST.COM (short username kudu)
I20260620 22:56:09.212890 25045 webserver.cc:460] Webserver started at http://127.2.90.189:36035/ using document root <none> and password file <none>
I20260620 22:56:09.213093 25045 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:09.213140 25045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:09.213263 25045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:09.214238 25045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data/instance:
uuid: "2b0e4958fb1540198df1f7f09f5f34bc"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "889df24d3f87a4a3f7544536ec86d087"
server_key_iv: "78e64e514b7ae62cfb44c54c3f6bc0eb"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.214550 25045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal/instance:
uuid: "2b0e4958fb1540198df1f7f09f5f34bc"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "889df24d3f87a4a3f7544536ec86d087"
server_key_iv: "78e64e514b7ae62cfb44c54c3f6bc0eb"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.215684 25045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 22:56:09.216275 25060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:09.216430 25045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:09.216506 25045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal
uuid: "2b0e4958fb1540198df1f7f09f5f34bc"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "889df24d3f87a4a3f7544536ec86d087"
server_key_iv: "78e64e514b7ae62cfb44c54c3f6bc0eb"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.216571 25045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:09.217648 25038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } attempt: 2
I20260620 22:56:09.218048 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.217911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40252)
0620 22:56:09.217969 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.218010 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 22:56:09.218354 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.218269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40268)
0620 22:56:09.218305 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.218332 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 22:56:09.218443 25038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.189:40635: Network error: Client connection negotiation failed: client connection to 127.2.90.189:40635: connect: Connection refused (error 111)
I20260620 22:56:09.226673 25045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:09.228340 25045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:09.228437 25045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:09.232024 25045 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.189:40635
I20260620 22:56:09.232052 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.189:40635 every 8 connection(s)
I20260620 22:56:09.232374 25045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/data/info.pb
I20260620 22:56:09.233306 25113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:09.234727 25113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:09.240375 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 25045
I20260620 22:56:09.240475 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-1/wal/instance
I20260620 22:56:09.240666 2410 external_mini_cluster.cc:1428] Setting key a2b7d86715ad8e89dd7e6f1cc6acfaad
I20260620 22:56:09.243010 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.235360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.2.90.190:36443)
0620 22:56:09.235415 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.235417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.235580 (+ 163us) server_negotiation.cc:407] Connection header received
0620 22:56:09.236016 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.236017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.236028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.236082 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.236668 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.237175 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.238102 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.238294 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.238343 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.240369 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.240372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.240372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.240379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.241745 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.242168 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.242169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.242170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.242225 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.242429 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.242430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.242437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.242547 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.242602 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.242842 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.242921 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 22:56:09.243176 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.235322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:36443 (local address 127.0.0.1:32870)
0620 22:56:09.235467 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.235506 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.235943 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.236143 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.236147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.236407 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.236592 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.236596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.237247 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.237248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.238038 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.238040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.238384 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.238385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.238504 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.238986 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.238994 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.240282 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.241831 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.241837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.241838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.242089 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.242279 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.242280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.242280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.242375 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.242600 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.242606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.242729 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.242812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.242968 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260620 22:56:09.243662 25113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.90.188@KRBTEST.COM; defaulting to no policy
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.90.188@KRBTEST.COM" created.
I20260620 22:56:09.250495 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.243892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40270)
0620 22:56:09.243939 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.243948 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.243986 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.244222 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.244222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.244319 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.244361 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.244362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.245572 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.245572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.246049 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.246050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.246633 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.246634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.246683 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.247009 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.247014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.248136 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.249574 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.249575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.249575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.249713 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.249891 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.249892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.249892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.249932 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.250175 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.250175 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.250214 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.250286 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.250370 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:56:09.250510 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.243920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.2.90.189:40635)
0620 22:56:09.244071 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.244076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.244100 (+ 24us) server_negotiation.cc:407] Connection header received
0620 22:56:09.244122 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.244124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.244138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.244220 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.244421 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.245488 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.246116 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.246565 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.246638 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.248205 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.248211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.248212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.248217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.249515 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.249769 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.249770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.249771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.249840 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.249969 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.249969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.249970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.250119 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.250194 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.250328 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.250414 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260620 22:56:09.251000 25113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:09.251266 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.251172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39864)
0620 22:56:09.251207 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.251246 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:56:09.251365 25061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
I20260620 22:56:09.251688 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.251611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39880)
0620 22:56:09.251647 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.251668 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:56:09.251775 25113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:41023: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.90.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab.
Entry for principal kudu/127.2.90.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab.
I20260620 22:56:09.261667 2410 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5TG4FZ/build/release/bin/kudu
/tmp/dist-test-task5TG4FZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:41023
--webserver_interface=127.2.90.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab
--principal=kudu/127.2.90.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.90.148:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.90.190:36443,127.2.90.189:40635,127.2.90.188:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 22:56:09.300988 25113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } attempt: 1
I20260620 22:56:09.301489 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.301345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39896)
0620 22:56:09.301407 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.301454 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 22:56:09.301828 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.301734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39910)
0620 22:56:09.301775 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.301805 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 22:56:09.301918 25113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:41023: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
I20260620 22:56:09.309562 25038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } attempt: 3
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:56:09.316263 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.309905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.2.90.189:40635)
0620 22:56:09.309950 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.309952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.309967 (+ 15us) server_negotiation.cc:407] Connection header received
0620 22:56:09.310037 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.310038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.310050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.310107 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.310369 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.311138 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.311788 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.312128 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.312191 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.314088 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.314089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.314089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.314095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.315326 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.315668 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.315669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.315669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.315711 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.315880 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.315881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.315881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.315974 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.316019 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.316154 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.316205 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 22:56:09.316318 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.309863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40278)
0620 22:56:09.309910 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.309921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.309977 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.310111 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.310112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.310246 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.310308 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.310310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.311207 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.311207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.311721 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.311723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.312193 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.312193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.312251 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.312637 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.312641 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.314014 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.315400 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.315401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.315402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.315607 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.315767 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.315768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.315768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.315832 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.316024 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.316024 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.316073 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.316135 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.316224 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 22:56:09.316709 25038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:09.317008 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.316910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39914)
0620 22:56:09.316953 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.316984 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 22:56:09.317293 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.317216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39930)
0620 22:56:09.317251 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.317276 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 22:56:09.317371 25038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:41023: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
W20260620 22:56:09.341063 25120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:56:09.341266 25120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:56:09.341284 25120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:56:09.341300 25120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:56:09.342384 25120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:56:09.342415 25120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 22:56:09.342427 25120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:56:09.342439 25120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:56:09.342454 25120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:56:09.343858 25120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.90.148:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.90.190:36443,127.2.90.189:40635,127.2.90.188:41023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/krb5kdc/kudu_127.2.90.188.keytab
--principal=kudu/127.2.90.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.90.188:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.90.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 22:56:09.344039 25120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:56:09.344206 25120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:56:09.348522 25125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:09.348512 25038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } attempt: 1
W20260620 22:56:09.348590 25126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:09.348639 25120 server_base.cc:1034] running on GCE node
W20260620 22:56:09.348800 25128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:56:09.348959 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.348836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39932)
0620 22:56:09.348884 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.348918 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 22:56:09.349058 25120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:56:09.349279 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.349210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39936)
0620 22:56:09.349244 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.349265 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 22:56:09.349359 25038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:41023: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
I20260620 22:56:09.349404 25120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:56:09.350558 25120 hybrid_clock.cc:648] HybridClock initialized: now 1781996169350545 us; error 23 us; skew 500 ppm
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 22:56:09.352849 25120 init.cc:377] Logged in from keytab as kudu/127.2.90.188@KRBTEST.COM (short username kudu)
I20260620 22:56:09.353205 25120 webserver.cc:460] Webserver started at http://127.2.90.188:42083/ using document root <none> and password file <none>
I20260620 22:56:09.353399 25120 fs_manager.cc:359] Metadata directory not provided
I20260620 22:56:09.353437 25120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:56:09.353525 25120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:56:09.354477 25120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data/instance:
uuid: "ccb71d8c59ea4e3587550fd0b221c322"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "5210bfe95022f7165d50b2d23f19f8e4"
server_key_iv: "177a7ea7ce6c7bd1b2e0fbc20060d665"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.354748 25120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal/instance:
uuid: "ccb71d8c59ea4e3587550fd0b221c322"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "5210bfe95022f7165d50b2d23f19f8e4"
server_key_iv: "177a7ea7ce6c7bd1b2e0fbc20060d665"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.355916 25120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 22:56:09.356582 25135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:56:09.356758 25120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 22:56:09.356835 25120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data,/tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal
uuid: "ccb71d8c59ea4e3587550fd0b221c322"
format_stamp: "Formatted at 2026-06-20 22:56:09 on dist-test-slave-4mtw"
server_key: "5210bfe95022f7165d50b2d23f19f8e4"
server_key_iv: "177a7ea7ce6c7bd1b2e0fbc20060d665"
server_key_version: "encryptionkey@0"
I20260620 22:56:09.356901 25120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:56:09.370045 25113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } attempt: 2
I20260620 22:56:09.370391 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.370278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39940)
0620 22:56:09.370329 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.370363 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 22:56:09.370685 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.370601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39954)
0620 22:56:09.370642 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.370666 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 22:56:09.370774 25113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.90.188:41023: Network error: Client connection negotiation failed: client connection to 127.2.90.188:41023: connect: Connection refused (error 111)
I20260620 22:56:09.372752 25120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:56:09.374604 25120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:56:09.374701 25120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:56:09.378317 25120 rpc_server.cc:307] RPC server started. Bound to: 127.2.90.188:41023
I20260620 22:56:09.378343 25187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.90.188:41023 every 8 connection(s)
I20260620 22:56:09.378638 25120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/data/info.pb
I20260620 22:56:09.379611 25188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:56:09.381101 25188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 22:56:09.385679 2410 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5TG4FZ/build/release/bin/kudu as pid 25120
I20260620 22:56:09.385768 2410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5TG4FZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781995912840861-2410-0/minicluster-data/master-2/wal/instance
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
I20260620 22:56:09.385950 2410 external_mini_cluster.cc:1428] Setting key 783a95c37a08dd3c777a98f81533d2ce
I20260620 22:56:09.388868 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.381698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.2.90.190:36443)
0620 22:56:09.381748 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.381750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.381900 (+ 150us) server_negotiation.cc:407] Connection header received
0620 22:56:09.382325 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.382325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.382333 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.382383 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.382910 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.383375 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.384240 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.384421 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.384462 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.386302 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.386304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.386304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.386311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.387683 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.388010 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.388011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.388011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.388065 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.388297 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.388298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.388298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.388396 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.388448 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.388753 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.388810 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 22:56:09.389079 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.381657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.190:36443 (local address 127.0.0.1:32884)
0620 22:56:09.381796 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.381833 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.382243 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.382450 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.382453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.382668 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.382843 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.382847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.383437 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.383438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.384185 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.384187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.384477 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.384478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.384580 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.384967 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.384975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.386221 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.387751 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.387756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.387757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.387953 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.388117 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.388117 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.388117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.388242 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.388448 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.388453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.388595 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.388735 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.388874 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260620 22:56:09.389604 25188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:56:09.394898 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.386269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.2.90.188:41023)
0620 22:56:09.386402 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.386407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.386446 (+ 39us) server_negotiation.cc:407] Connection header received
0620 22:56:09.386482 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.386485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.386497 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.386565 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.387179 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.388086 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.389077 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.389521 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.389577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.392043 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.392050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.392050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.392057 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.393765 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.394114 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.394117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.394118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.394173 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.394351 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.394352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.394352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.394504 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.394631 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.394721 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.394790 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260620 22:56:09.396596 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.389884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.2.90.189:40635)
0620 22:56:09.389923 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.389924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.390125 (+ 201us) server_negotiation.cc:407] Connection header received
0620 22:56:09.390185 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.390186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.390193 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.390233 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.390475 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.391069 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.392206 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.392518 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.392575 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.394371 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.394373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.394373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.394380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.395745 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.396013 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.396014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.396014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.396071 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.396218 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.396219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.396219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.396316 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.396358 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.396490 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.396541 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:56:09.396633 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.389821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.189:40635 (local address 127.0.0.1:40280)
0620 22:56:09.390073 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.390083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.390130 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.390237 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.390238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.390361 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.390415 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.390416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.391130 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.391131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.392127 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.392129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.392599 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.392600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.392700 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.393156 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.393161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.394300 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.395808 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.395809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.395810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.395952 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.396127 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.396127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.396127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.396171 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.396367 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.396367 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.396404 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.396472 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.396549 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260620 22:56:09.396955 25188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.190@KRBTEST.COM
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.188@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:56:09.402349 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.395614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.2.90.190:36443)
0620 22:56:09.395647 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.395649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.395656 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:09.395694 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.395695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.395702 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.395740 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.396183 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.396829 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.397687 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.397979 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.398040 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.400055 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.400056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.400056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.400063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.401440 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.401714 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.401715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.401716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.401766 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.401899 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.401899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.401900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.402020 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.402127 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.402200 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.402265 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 22:56:09.402858 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.397173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39974)
0620 22:56:09.397207 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.397215 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.397242 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.397338 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.397339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.397463 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.397492 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.397493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.398211 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.398211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.398904 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.398906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.399227 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.399228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.399282 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.399580 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.399584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.400565 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.401967 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.401967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.401968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.402123 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.402301 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.402302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.402302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.402368 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.402599 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.402599 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.402641 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.402705 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.402790 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 22:56:09.402938 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.397208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.2.90.188:41023)
0620 22:56:09.397237 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.397239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.397246 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:09.397288 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.397289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.397296 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.397334 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.397544 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.398147 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.398977 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.399163 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.399211 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.400650 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.400650 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.400651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.400657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.401902 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.402188 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.402189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.402189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.402241 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.402424 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.402425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.402425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.402529 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.402589 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.402738 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.402844 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":14}
I20260620 22:56:09.403973 25188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: Bootstrap starting.
I20260620 22:56:09.404724 25188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:09.404973 25188 log.cc:824] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:09.405701 25188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: No bootstrap required, opened a new log
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.90.189@KRBTEST.COM
I20260620 22:56:09.407434 25188 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.407577 25188 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb71d8c59ea4e3587550fd0b221c322, State: Initialized, Role: FOLLOWER
I20260620 22:56:09.407662 25188 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.407941 25188 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:09.407919 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.408027 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:09.409149 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.402929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.2.90.189:40635)
0620 22:56:09.402958 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.402960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.403029 (+ 69us) server_negotiation.cc:407] Connection header received
0620 22:56:09.403108 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.403109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.403118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.403161 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.403561 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.403979 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.404563 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.404817 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.404871 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.406903 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.406904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.406904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.406911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.408205 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.408492 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.408493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.408493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.408542 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.408725 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.408726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.408726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.408821 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.408944 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.409010 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.409068 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 22:56:09.409596 25205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:09.409643 25205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:09.410534 2410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 22:56:09.409874 25120 master_runner.cc:186] Error getting master registration for 127.2.90.190:36443: OK, instance_id { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" instance_seqno: 1781996169085450 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:56:09.412264 25120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:56:09.411643 25012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "358b3abe3b414490871f0e98aed06f55"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32892
I20260620 22:56:09.424474 25038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } attempt: 2
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.190@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:56:09.431607 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.424771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:39990)
0620 22:56:09.424822 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.424833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.424883 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.425000 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.425001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.425173 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.425221 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.425224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.426087 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.426088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.426891 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.426893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.427321 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.427322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.427369 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.427800 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.427804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.428999 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.430625 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.430626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.430626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.430828 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.430999 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.431000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.431000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.431057 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.431322 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.431322 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.431364 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.431426 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.431507 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 22:56:09.431620 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.424850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.2.90.188:41023)
0620 22:56:09.424890 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.424892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.424901 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:09.424923 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.424924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.424933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.424980 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.425289 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.426021 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.426964 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.427261 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.427325 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.429076 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.429078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.429078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.429085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.430541 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.430890 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.430891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.430892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.430945 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.431107 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.431108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.431108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.431264 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.431328 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.431456 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.431528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 22:56:09.433147 25038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55: Bootstrap starting.
I20260620 22:56:09.433810 25038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:09.434077 25038 log.cc:824] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:09.434752 25038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55: No bootstrap required, opened a new log
I20260620 22:56:09.435702 25038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.435811 25038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358b3abe3b414490871f0e98aed06f55, State: Initialized, Role: FOLLOWER
I20260620 22:56:09.435885 25038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.436108 25208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.436169 25038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:09.436183 25208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:09.436285 25012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:56:09.436388 25012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:09.437183 25012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.437749 25012 leader_election.cc:288] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0e4958fb1540198df1f7f09f5f34bc (127.2.90.189:40635), ccb71d8c59ea4e3587550fd0b221c322 (127.2.90.188:41023)
W20260620 22:56:09.438122 25219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:09.438174 25219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:09.438172 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "358b3abe3b414490871f0e98aed06f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ccb71d8c59ea4e3587550fd0b221c322"
I20260620 22:56:09.438280 25162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:09.439138 25162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 358b3abe3b414490871f0e98aed06f55 in term 1.
I20260620 22:56:09.439355 24986 leader_election.cc:302] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 358b3abe3b414490871f0e98aed06f55, ccb71d8c59ea4e3587550fd0b221c322; no voters:
I20260620 22:56:09.439457 25208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:56:09.439574 25208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [term 1 LEADER]: Becoming Leader. State: Replica: 358b3abe3b414490871f0e98aed06f55, State: Running, Role: LEADER
I20260620 22:56:09.439646 25208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.439998 25208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358b3abe3b414490871f0e98aed06f55. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.440069 25208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:09.440217 25223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:56:09.440368 25223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:56:09.438669 24970 master_runner.cc:186] Error getting master registration for 127.2.90.189:40635: OK, instance_id { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" instance_seqno: 1781996169231038 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 22:56:09.440845 24970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 22:56:09.442714 25162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [term 1 FOLLOWER]: Refusing update from remote peer 358b3abe3b414490871f0e98aed06f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:09.442986 25208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:09.444245 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358b3abe3b414490871f0e98aed06f55. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.444326 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:09.445149 25228 mvcc.cc:205] Tried to move back new op lower bound from 7299056310034841600 to 7299056310025322496. Current Snapshot: MvccSnapshot[applied={T|T < 7299056310034841600}]
I20260620 22:56:09.445356 25208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.445437 25208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: This master's current role is: LEADER
I20260620 22:56:09.445770 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.445845 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:09.446478 25229 mvcc.cc:205] Tried to move back new op lower bound from 7299056310034841600 to 7299056310025322496. Current Snapshot: MvccSnapshot[applied={T|T < 7299056310034841600}]
I20260620 22:56:09.449000 25223 catalog_manager.cc:1347] Generated new cluster ID: c2890f84dd964c53bbe1306ebdc20a7a
I20260620 22:56:09.449043 25223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:56:09.461905 25113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } attempt: 3
I20260620 22:56:09.466137 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.443202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.2.90.190:36443)
0620 22:56:09.443237 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.443238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.444238 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 22:56:09.444466 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.444467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.444488 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.444515 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.445854 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.446555 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.447746 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.448053 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.448117 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.462962 (+ 14845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.462963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.462963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.462971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.464357 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.464756 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.464758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.464758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.464825 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.465536 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.465537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.465537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.465669 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.465754 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.465951 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.466042 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:09.468572 25223 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:56:09.469332 25223 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:56:09.472931 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.459228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.2.90.188:41023)
0620 22:56:09.459264 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.459266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.459591 (+ 325us) server_negotiation.cc:407] Connection header received
0620 22:56:09.459751 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.459752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.459777 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.459804 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.461224 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.461950 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.464276 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.464576 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.464611 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.466874 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.466875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.466875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.466882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.471147 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.471777 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.471778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.471778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.471850 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.472150 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.472151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.472151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.472306 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.472388 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.472766 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.472843 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 22:56:09.475059 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.443722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.2.90.189:40635)
0620 22:56:09.443762 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.443764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.447847 (+ 4083us) server_negotiation.cc:407] Connection header received
0620 22:56:09.447936 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.447937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.447948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.447975 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.448625 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.449669 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.460887 (+ 11218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.461210 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.461250 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.467284 (+ 6034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.467285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.467285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.467293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.470272 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.473957 (+ 3685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.473958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.473959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.474024 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.474501 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.474502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.474503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.474633 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.474706 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.474893 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.474968 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 22:56:09 dist-test-slave-4mtw krb5kdc[24957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996169, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.90.189@KRBTEST.COM for kudu/127.2.90.188@KRBTEST.COM
I20260620 22:56:09.479382 25223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55: Generated new TSK 0
I20260620 22:56:09.479596 25223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:56:09.481567 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.462145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.90.188:41023 (local address 127.0.0.1:40004)
0620 22:56:09.462272 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 22:56:09.462283 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 22:56:09.462321 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 22:56:09.468034 (+ 5713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 22:56:09.468035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 22:56:09.468166 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 22:56:09.468207 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.468209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.474351 (+ 6142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.474352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.475165 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 22:56:09.475167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.476194 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.476195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 22:56:09.476272 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.476749 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 22:56:09.476768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 22:56:09.478154 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 22:56:09.480491 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.480493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.480493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.480720 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.480940 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 22:56:09.480941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 22:56:09.480942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 22:56:09.481009 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 22:56:09.481275 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 22:56:09.481275 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 22:56:09.481317 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 22:56:09.481380 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 22:56:09.481472 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 22:56:09.481580 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:09.462407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.2.90.188:41023)
0620 22:56:09.467577 (+ 5170us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.467578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.467585 (+ 7us) server_negotiation.cc:407] Connection header received
0620 22:56:09.467602 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.467603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.467636 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.468275 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.468320 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.474272 (+ 5952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.475545 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:09.476105 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:09.476174 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:09.478235 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 22:56:09.478236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 22:56:09.478236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 22:56:09.478242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 22:56:09.480315 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.480795 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.480796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.480796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.480859 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 22:56:09.481082 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 22:56:09.481083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 22:56:09.481083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 22:56:09.481217 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 22:56:09.481273 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:09.481415 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 22:56:09.481486 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5149,"thread_start_us":66,"threads_started":1}
I20260620 22:56:09.483155 25113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: Bootstrap starting.
I20260620 22:56:09.483865 25113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: Neither blocks nor log segments found. Creating new log.
I20260620 22:56:09.484131 25113 log.cc:824] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: Log is configured to *not* fsync() on all Append() calls
I20260620 22:56:09.485270 25113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: No bootstrap required, opened a new log
I20260620 22:56:09.488166 25113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.488300 25113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e4958fb1540198df1f7f09f5f34bc, State: Initialized, Role: FOLLOWER
I20260620 22:56:09.488384 25113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } }
I20260620 22:56:09.488710 25113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:56:09.488759 25086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:56:09.489570 25086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [term 1 FOLLOWER]: Refusing update from remote peer 358b3abe3b414490871f0e98aed06f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:56:09.489730 25087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "358b3abe3b414490871f0e98aed06f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2b0e4958fb1540198df1f7f09f5f34bc"
I20260620 22:56:09.490412 25087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 358b3abe3b414490871f0e98aed06f55 in term 1.
I20260620 22:56:09.490808 25234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 22:56:09.492515 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358b3abe3b414490871f0e98aed06f55. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.492604 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:09.492657 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.492755 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: This master's current role is: FOLLOWER
W20260620 22:56:09.493244 25251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 22:56:09.493335 25251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 22:56:09.493413 25240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.493526 25240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 22:56:09.494665 25253 mvcc.cc:205] Tried to move back new op lower bound from 7299056310181400576 to 7299056310025322496. Current Snapshot: MvccSnapshot[applied={T|T < 7299056310034841600}]
W20260620 22:56:09.511624 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.510421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.2.90.189:40635)
0620 22:56:09.510461 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.510462 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.511029 (+ 567us) server_negotiation.cc:407] Connection header received
0620 22:56:09.511121 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.511122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.511132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.511158 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.511566 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40306: BlockingRecv error: recv got EOF from 127.0.0.1:40306 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:56:09.512326 25041 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.510766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.2.90.190:36443)
0620 22:56:09.510802 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.510803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.511844 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 22:56:09.511931 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.511932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.511941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.511967 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.512259 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32906: BlockingRecv error: recv got EOF from 127.0.0.1:32906 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 22:56:09.513280 25237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.512636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.2.90.188:41023)
0620 22:56:09.512672 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.512674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.512810 (+ 136us) server_negotiation.cc:407] Connection header received
0620 22:56:09.512895 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.512896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.512904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.512930 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.513226 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40020: BlockingRecv error: recv got EOF from 127.0.0.1:40020 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:56:09.515249 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.514568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.2.90.189:40635)
0620 22:56:09.514598 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.514599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.514808 (+ 209us) server_negotiation.cc:407] Connection header received
0620 22:56:09.514899 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.514900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.514915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.514941 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.515209 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40314: BlockingRecv error: recv got EOF from 127.0.0.1:40314 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 22:56:09.517956 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.517285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.2.90.189:40635)
0620 22:56:09.517326 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.517328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.517542 (+ 214us) server_negotiation.cc:407] Connection header received
0620 22:56:09.517622 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.517623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.517631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.517656 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.517917 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40330: BlockingRecv error: recv got EOF from 127.0.0.1:40330 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 22:56:09.521587 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.520913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.2.90.189:40635)
0620 22:56:09.520941 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.520943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.521159 (+ 216us) server_negotiation.cc:407] Connection header received
0620 22:56:09.521241 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.521242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.521250 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.521275 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.521549 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40346: BlockingRecv error: recv got EOF from 127.0.0.1:40346 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 22:56:09.527169 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.526498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.2.90.189:40635)
0620 22:56:09.526528 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.526529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.526732 (+ 203us) server_negotiation.cc:407] Connection header received
0620 22:56:09.526817 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.526818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.526826 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.526853 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.527128 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40354: BlockingRecv error: recv got EOF from 127.0.0.1:40354 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 22:56:09.536938 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.536154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.2.90.189:40635)
0620 22:56:09.536187 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.536189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.536418 (+ 229us) server_negotiation.cc:407] Connection header received
0620 22:56:09.536506 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.536507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.536516 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.536542 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.536879 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40358: BlockingRecv error: recv got EOF from 127.0.0.1:40358 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 22:56:09.554898 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.554061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.2.90.189:40635)
0620 22:56:09.554100 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.554102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.554371 (+ 269us) server_negotiation.cc:407] Connection header received
0620 22:56:09.554467 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.554468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.554478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.554506 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.554844 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: recv got EOF from 127.0.0.1:40360 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 22:56:09.589200 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.588197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.2.90.189:40635)
0620 22:56:09.588243 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.588246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.588570 (+ 324us) server_negotiation.cc:407] Connection header received
0620 22:56:09.588672 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.588692 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.588704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.588735 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.589140 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40370: BlockingRecv error: recv got EOF from 127.0.0.1:40370 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 22:56:09.655131 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.654493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.2.90.189:40635)
0620 22:56:09.654557 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.654559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.654677 (+ 118us) server_negotiation.cc:407] Connection header received
0620 22:56:09.654711 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.654711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.654724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.654780 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.655067 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40382: BlockingRecv error: recv got EOF from 127.0.0.1:40382 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 22:56:09.784883 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:09.784228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.2.90.189:40635)
0620 22:56:09.784304 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:09.784306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:09.784353 (+ 47us) server_negotiation.cc:407] Connection header received
0620 22:56:09.784423 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:09.784424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:09.784445 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:09.784492 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:09.784822 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40398: BlockingRecv error: recv got EOF from 127.0.0.1:40398 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 22:56:09.927387 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "358b3abe3b414490871f0e98aed06f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358b3abe3b414490871f0e98aed06f55" member_type: VOTER last_known_addr { host: "127.2.90.190" port: 36443 } } peers { permanent_uuid: "2b0e4958fb1540198df1f7f09f5f34bc" member_type: VOTER last_known_addr { host: "127.2.90.189" port: 40635 } } peers { permanent_uuid: "ccb71d8c59ea4e3587550fd0b221c322" member_type: VOTER last_known_addr { host: "127.2.90.188" port: 41023 } } }
I20260620 22:56:09.927536 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358b3abe3b414490871f0e98aed06f55 [sys.catalog]: This master's current role is: LEADER
W20260620 22:56:10.042479 25117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 22:56:10.041910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.2.90.189:40635)
0620 22:56:10.041981 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:10.041983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:10.041992 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:10.042051 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:10.042052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:10.042064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:10.042110 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 22:56:10.042418 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40408: BlockingRecv error: recv got EOF from 127.0.0.1:40408 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 22:56:10.410046 25205 catalog_manager.cc:1259] Loaded cluster ID: c2890f84dd964c53bbe1306ebdc20a7a
I20260620 22:56:10.410113 25205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: loading cluster ID for follower catalog manager: success
I20260620 22:56:10.410898 25205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: acquiring CA information for follower catalog manager: success
I20260620 22:56:10.411192 25205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccb71d8c59ea4e3587550fd0b221c322: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:56:10.493772 25251 catalog_manager.cc:1259] Loaded cluster ID: c2890f84dd964c53bbe1306ebdc20a7a
I20260620 22:56:10.493844 25251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: loading cluster ID for follower catalog manager: success
I20260620 22:56:10.494637 25251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: acquiring CA information for follower catalog manager: success
I20260620 22:56:10.494915 25251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b0e4958fb1540198df1f7f09f5f34bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 22:56:10.557844 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:10.555443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.2.90.189:40635)
0620 22:56:10.555569 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:10.555571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:10.555580 (+ 9us) server_negotiation.cc:407] Connection header received
0620 22:56:10.555643 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:10.555643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:10.555691 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:10.555762 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:10.555871 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.556384 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.557149 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.557356 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.557405 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:10.557555 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:10.557666 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:10.557734 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:10.557750 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 22:56:10.557771 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260620 22:56:10.561813 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:10.559469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.2.90.188:41023)
0620 22:56:10.559568 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:10.559570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:10.559578 (+ 8us) server_negotiation.cc:407] Connection header received
0620 22:56:10.559649 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:10.559650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:10.559697 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:10.559760 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:10.559853 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.560342 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.561113 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.561320 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.561371 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:10.561511 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:10.561612 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:10.561675 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:10.561714 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 22:56:10.561734 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":43,"threads_started":1}
I20260620 22:56:10.565752 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:10.563202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.2.90.189:40635)
0620 22:56:10.563288 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:10.563290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:10.563388 (+ 98us) server_negotiation.cc:407] Connection header received
0620 22:56:10.563533 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:10.563534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:10.563541 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:10.563580 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:10.563685 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.564189 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.565044 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.565305 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.565358 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:10.565502 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:10.565533 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:10.565573 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:10.565664 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 22:56:10.565685 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 22:56:10.565866 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:10.563181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.2.90.188:41023)
0620 22:56:10.563214 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:10.563216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:10.563336 (+ 120us) server_negotiation.cc:407] Connection header received
0620 22:56:10.563437 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:10.563439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:10.563448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:10.563494 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:10.563604 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.564301 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.565161 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.565435 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.565468 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:10.565622 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:10.565648 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:10.565689 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:10.565782 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 22:56:10.565802 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 22:56:10.566306 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 22:56:10.563325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.2.90.190:36443)
0620 22:56:10.563549 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 22:56:10.563551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 22:56:10.563557 (+ 6us) server_negotiation.cc:407] Connection header received
0620 22:56:10.563603 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 22:56:10.563604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 22:56:10.563650 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 22:56:10.563745 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 22:56:10.563840 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.564545 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.565367 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 22:56:10.565689 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 22:56:10.565887 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 22:56:10.565954 (+ 67us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 22:56:10.566056 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 22:56:10.566136 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 22:56:10.566192 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 22:56:10.566226 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":44,"threads_started":1}
I20260620 22:56:10.567730 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 24970
I20260620 22:56:10.572515 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 25045
I20260620 22:56:10.576663 2410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5TG4FZ/build/release/bin/kudu with pid 25120
2026-06-20T22:56:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2651 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5303 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (257741 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST